Skip to content

feat(api): add sandbox fork endpoint#3202

Open
mishushakov wants to merge 14 commits into
mainfrom
sandbox-fork-api-method
Open

feat(api): add sandbox fork endpoint#3202
mishushakov wants to merge 14 commits into
mainfrom
sandbox-fork-api-method

Conversation

@mishushakov

@mishushakov mishushakov commented Jul 3, 2026

Copy link
Copy Markdown
Member

Summary

  • Adds POST /sandboxes/{sandboxID}/fork: checkpoints the running sandbox in place and creates one or more new sandboxes from that snapshot under fresh IDs.
  • The checkpoint reuses the mechanism behind snapshot templates (Sandbox.Checkpoint gRPC): the sandbox is briefly paused on its node, snapshotted to its own snapshots row, and resumed under the same execution ID — the original never leaves the sandbox store, so its ID, expiration, node placement, and concurrency reservation are untouched.
  • SandboxForkRequest supports an optional timeout (TTL for the forked sandboxes, defaults to the standard 15s sandbox default) and count (number of forks, default 1). The snapshot is captured once regardless of count; all forks boot from it in parallel.
  • Per-fork results: the 201 response is a list of SandboxForkResult, each holding either the created sandbox or the error that prevented it from starting (Promise.allSettled-style). A non-201 status means the request failed before any fork was attempted. Checkpoint always captures full memory state; forking requires a snapshot-capable envd (400 otherwise).
  • Splits the resume data fetcher into buildResumeSandboxData (resume from own snapshot) and buildResumeSandboxDataFromSnapshot (boot a different ID from a snapshot), so each forked sandbox's envd access token is generated for the ID it actually runs under.
  • Threads SnapshotSandboxID through SandboxMetadata so the resume origin-node remap on placement timeout targets the snapshot's row (previously a silent no-op when the started ID differed from the snapshot's).
  • Regenerated packages/api/internal/api/api.gen.go and the integration test client from the updated OpenAPI spec.

Test plan

  • go build/go vet/go test -race pass for packages/api/... and tests/integration
  • make fmt and scoped golangci-lint run on touched packages are clean
  • Integration tests: happy-path fork (original and fork both running afterwards), multi-fork with count: 2 (unique IDs, all running, no per-fork errors), invalid count: 0 (400), forking a paused sandbox (409), forking a killed sandbox (404), cross-team access (404)

🤖 Generated with Claude Code

Adds POST /sandboxes/{sandboxID}/fork, which snapshots a running sandbox,
resumes the original under its original ID (preserving its remaining TTL),
and creates a new sandbox from the same snapshot with its own timeout.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

@cursor

cursor Bot commented Jul 3, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
New sandbox lifecycle path reuses snapshot/checkpoint gRPC and can kill the source on checkpoint failure; parallel forks up to 100 interact with team concurrency and placement limits.

Overview
This PR exposes forking a running sandbox via POST /sandboxes/{sandboxID}/fork. The handler checkpoints the source in place (pause, full-memory snapshot to that sandbox’s snapshot row, resume under the same ID so TTL and concurrency stay unchanged), then starts up to count new sandboxes in parallel from that snapshot, each with its own ID and optional TTL. The 201 body is a per-fork result list (sandbox or error), while 4xx/5xx mean nothing was forked. Resume-from-snapshot wiring now separates which ID owns the snapshot from which ID is being started, including envd tokens and origin-node remap on placement timeout. OpenAPI and generated API clients are updated accordingly, with integration coverage for the new flow.

Reviewed by Cursor Bugbot for commit bf16ec1. Bugbot is set up for automated code reviews on this repo. Configure here.

@codecov

codecov Bot commented Jul 3, 2026

Copy link
Copy Markdown

❌ 2 Tests Failed:

Tests completed Failed Passed Skipped
3174 2 3172 6
View the top 3 failed test(s) by shortest run time
github.com/e2b-dev/infra/packages/api/internal/sandbox/storage/redis::TestStorageLocker_ObtainUsesProvidedContext
Stack Traces | 14.7s run time
=== RUN   TestStorageLocker_ObtainUsesProvidedContext
=== PAUSE TestStorageLocker_ObtainUsesProvidedContext
=== CONT  TestStorageLocker_ObtainUsesProvidedContext
    lock_test.go:97: 
        	Error Trace:	.../storage/redis/lock_test.go:289
        	            				.../storage/redis/lock_test.go:97
        	Error:      	Received unexpected error:
        	            	context deadline exceeded
        	Test:       	TestStorageLocker_ObtainUsesProvidedContext
--- FAIL: TestStorageLocker_ObtainUsesProvidedContext (14.67s)
github.com/e2b-dev/infra/tests/integration/internal/tests/orchestrator::TestHardwareEntropyDeviceAvailability
Stack Traces | 18.7s run time
=== RUN   TestHardwareEntropyDeviceAvailability
=== PAUSE TestHardwareEntropyDeviceAvailability
=== CONT  TestHardwareEntropyDeviceAvailability
    sandbox_entropy_test.go:25: Command [bash] output: event:{start:{pid:1138}}
    sandbox_entropy_test.go:26: 
        	Error Trace:	.../tests/orchestrator/sandbox_entropy_test.go:26
        	Error:      	Received unexpected error:
        	            	failed to execute command bash in sandbox i7jsrxuab569adoz83c9s: invalid_argument: protocol error: incomplete envelope: unexpected EOF
        	Test:       	TestHardwareEntropyDeviceAvailability
--- FAIL: TestHardwareEntropyDeviceAvailability (18.74s)
github.com/e2b-dev/infra/tests/integration/internal/tests/api/sandboxes::TestSandboxConnect_FilesystemOnlyResumes
Stack Traces | 61.5s run time
=== RUN   TestSandboxConnect_FilesystemOnlyResumes
=== PAUSE TestSandboxConnect_FilesystemOnlyResumes
=== CONT  TestSandboxConnect_FilesystemOnlyResumes
Executing command cat in sandbox i19uvq6mn0d1opxd9z486 (user: root)
    filesystem_only_test.go:92: 
        	Error Trace:	.../api/sandboxes/filesystem_only_test.go:92
        	Error:      	Not equal: 
        	            	expected: 201
        	            	actual  : 500
        	Test:       	TestSandboxConnect_FilesystemOnlyResumes
        	Messages:   	connecting to a paused filesystem-only snapshot should resume it (cold boot)
--- FAIL: TestSandboxConnect_FilesystemOnlyResumes (61.46s)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

In packages/api/internal/handlers/sandbox_fork.go, the parsed request body returned by ParseOptionalBody can be nil if the request body is omitted. Accessing body.Timeout and body.Memory without a nil check will cause a runtime panic, so a nil check must be added before dereferencing these fields.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread packages/api/internal/handlers/sandbox_fork.go Outdated
Comment thread packages/api/internal/handlers/sandbox_fork.go Outdated
Comment thread packages/api/internal/handlers/sandbox_fork.go Outdated
Comment thread packages/api/internal/handlers/sandbox_fork.go Outdated
Comment thread packages/api/internal/handlers/sandbox_fork.go
Comment thread packages/api/internal/handlers/sandbox_fork.go Outdated
Comment thread packages/api/internal/handlers/sandbox_fork.go
mishushakov and others added 2 commits July 3, 2026 13:58
Runs the original sandbox's resume and the new forked sandbox's creation in
parallel instead of sequentially, since both boot independently from the
same immutable snapshot. Errors from either are still surfaced to the
caller; if the original fails to resume, the orphaned fork is cleaned up.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- generate the envd access token for the ID the sandbox runs under, so
  forked secure sandboxes don't get a token keyed to the original ID
- recompute the original sandbox's remaining TTL after the pause so the
  resume preserves its expiration instead of extending it by the pause
  duration
- only treat GetSandbox not-found as "not running"; other errors are 500
- best-effort team concurrency check before pausing so fork doesn't
  pause the original only to fail creating the second sandbox
- name the resume recovery path in the error when the original fails to
  resume after a successful pause

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment thread packages/api/internal/handlers/sandbox_fork.go Outdated
mishushakov and others added 2 commits July 3, 2026 14:50
When a fork placement times out, the remap ran with the forked sandbox
ID, which has no snapshot row: the DB update matched nothing and the
wrong cache key was invalidated, while still logging and counting a
remap. Thread the snapshot's sandbox ID through SandboxMetadata so the
remap targets the row the resume actually reads.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
buildResumeSandboxData keeps its original one-ID signature for the
common resume-from-own-snapshot case and delegates to the new
buildResumeSandboxDataFromSnapshot, which fork uses to boot a new ID
from the original sandbox's snapshot.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment thread packages/api/internal/handlers/sandbox_fork.go Outdated
mishushakov and others added 5 commits July 3, 2026 15:31
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-fork

Instead of granting the expired original a fresh default timeout, leave
it paused: it can still be restored with the resume endpoint, and its
lifetime is never extended past the expiration it had before the fork.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e/resume

Rebase the fork endpoint on the checkpoint mechanism used by snapshot
templates: the sandbox is briefly paused on its node, snapshotted to its
own snapshots row, and resumed under the same execution ID. The original
never leaves the sandbox store, so its ID, expiration, and concurrency
reservation are untouched and the whole pause/resume failure surface
(TTL drift, expired-mid-pause, resume rollback, concurrency pre-check)
disappears. The fork then boots from that snapshot under a new ID.

Checkpoint requires a memory snapshot and a snapshot-capable envd, so
the request's memory (filesystem-only) option is removed and forking an
old-envd sandbox returns 400.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The snapshot is captured once and all forks boot from it in parallel.
The 201 response is now a list of sandboxes. Creation is all-or-nothing:
if any fork fails to start, the ones that did are killed and the error
is returned, so callers never reconcile a partial result. A count at or
above the team concurrency limit fails fast with 400 since the original
sandbox keeps holding one slot.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment thread packages/api/internal/handlers/sandbox_fork.go Fixed
Comment thread packages/api/internal/handlers/sandbox_fork.go Fixed
Each fork now succeeds or fails independently: the 201 response carries
one SandboxForkResult per requested fork, holding either the created
sandbox or the error that prevented it from starting. Callers map the
outcomes like Promise.allSettled instead of retrying the whole batch
when one fork hits a limit. A non-201 status still means the request
failed before any fork was attempted.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment thread packages/api/internal/handlers/sandbox_fork.go
The concurrency fail-fast only ran when the request set count, so an
omitted count (default 1) with a concurrency limit of 1 checkpointed the
sandbox and returned a per-fork error where an explicit count returned
400. Also cap count at 100 per request, bounding the parallel boots and
the user-sized result allocation flagged by CodeQL.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment thread packages/api/internal/orchestrator/checkpoint_instance.go
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment thread packages/api/internal/orchestrator/checkpoint_instance.go
…e pausing

The orchestrator rejects a checkpoint with FailedPrecondition (envd too
old) or ResourceExhausted (starting-sandboxes queue full) before it ever
pauses the VM, so the sandbox is still running healthy on its node.
CheckpointSandbox killed it anyway on any RPC error; now those
rejections restore the sandbox to Running, and queue exhaustion surfaces
as a retriable 503 from the fork endpoint, mirroring the pause flow's
PauseQueueExhaustedError handling.

The failure cleanup (failSnapshotBuild, RemoveSandbox) also ran on the
cancellable request context, so a client disconnect mid-checkpoint made
the cleanup itself fail and stranded the sandbox in Snapshotting for its
full remaining TTL; run it on context.WithoutCancel like the neighboring
finishSnapshotting call.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit bf16ec1. Configure here.

Comment thread tests/integration/internal/tests/api/sandboxes/sandbox_fork_test.go
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants