nfsproxy: make NFS file-handle cache limit configurable via NFS_PROXY_CACHE_LIMIT - #3549
Open
AdaAibaby wants to merge 4 commits into
Open
nfsproxy: make NFS file-handle cache limit configurable via NFS_PROXY_CACHE_LIMIT#3549AdaAibaby wants to merge 4 commits into
AdaAibaby wants to merge 4 commits into
Conversation
Fixes e2b-dev#3546. The nbd kernel module is loaded with nbds_max=4096, pre-creating 4096 /dev/nbdX block devices on every sandbox node. `lsblk | grep nbd` produces 4096 lines of noise even when zero sandboxes are running, making it impossible to quickly identify which devices are actively connected. ConnectedDevices() in pool.go already scans /sys/block/nbdX/pid to find connected slots, but was not exposed via any operator-facing tool. Changes: - Add cmd/inspect-nbd: reads ConnectedDevices() and prints a table of active NBD slots with device path, size, and PID. Supports -json for machine-readable output. Linux-only (//go:build linux). - Add DevicePool.maxDevices field: stored at construction to avoid re-reading /sys/module/nbd/parameters/nbds_max on each call. - Add DevicePool.Status() -> PoolStatus: returns Max/Used/PreWarmed counts from the in-memory bitset without scanning sysfs — usable by future debug endpoints. Example output on a node with 4096 configured but only 1 connected: NBD devices: 1 connected / 4096 configured SLOT DEVICE SIZE (MB) PID 1 /dev/nbd1 22691 3321246
…_CACHE_LIMIT Fixes e2b-dev#3548. The CachingHandler LRU was hard-coded to 1024 entries. When a sandbox installs a large package (e.g. `npm i -g @openai/codex`, which fetches 6 platform-specific optional packages simultaneously), the cache fills and oldest entries are evicted. Subsequent RPCs using evicted handles return NFS3ERR_STALE, which Linux translates to ESTALE (errno 116), surfacing as `npm error errno -116 UNKNOWN: unknown error, write`. Additionally, the 1024-slot cache was shared across all sandboxes on the node, so every concurrent sandbox competed for the same slots. Changes: - Remove `const cacheLimit = 1024` from proxy.go - Add `NFSProxyCacheLimit int` to orchestrator cfg/model.go with env:"NFS_PROXY_CACHE_LIMIT" envDefault:"16384" (16384 entries ≈ 2.4 MB — negligible overhead) - Add `CacheLimit int` to nfsproxy/cfg/model.go - Pass config.NFSProxyCacheLimit through run.go → nfscfg.Config → NewProxy Operators can now tune the cache size at runtime by setting NFS_PROXY_CACHE_LIMIT in the Nomad job env without recompiling.
AdaAibaby
requested review from
ValentaTomas,
dobrac and
jakubno
as code owners
August 7, 2026 08:56
|
We encountered the same issue. As a temporary workaround, we increased the cache size to 16 * 1024. We later tried switching to a per-sandbox cache, but found that it increased sandbox creation time. We have not yet identified the exact cause. |
leonmeijer
reviewed
Aug 16, 2026
leonmeijer
left a comment
leonmeijer
left a comment
There was a problem hiding this comment.
NFS_PROXY_CACHE_LIMIT accepts zero or negative values and NewProxy passes them directly to helpers.NewCachingHandler. The pinned go-nfs helper ignores the LRU constructor errors, leaves nil caches, and then panics on the first handle operation; zero-value Config callers have the same failure. Validate a positive limit and give nfsproxy.Config a safe default before constructing the handler.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3548.
Problem
CachingHandlerin go-nfs maps NFS file handles (UUID) to(filesystem, path)via an LRU. The LRU size was hardcoded atconst cacheLimit = 1024. When a sandbox installs a large package — for examplenpm i -g @openai/codex, which pulls 6 platform-specific optional packages simultaneously — the cache fills and oldest entries are evicted. Any subsequent RPC using an evicted handle returnsNFS3ERR_STALE, which Linux translates toESTALE(errno 116):The problem is amplified by the fact that the single 1024-slot LRU was shared across all sandboxes on the node.
Fix
Replace the hardcoded constant with a configurable env var so operators can tune the cache without recompiling the orchestrator:
const cacheLimit = 1024(hardcoded)NFS_PROXY_CACHE_LIMIT(env var, default16384)Changes
pkg/cfg/model.go: addNFSProxyCacheLimit intwithenv:"NFS_PROXY_CACHE_LIMIT" envDefault:"16384"pkg/nfsproxy/cfg/model.go: addCacheLimit intfieldpkg/nfsproxy/proxy.go: removeconst cacheLimit = 1024; useconfig.CacheLimitpkg/factories/run.go: threadconfig.NFSProxyCacheLimit→nfscfg.Config.CacheLimitNot in scope
Per-sandbox cache isolation (each mount gets its own
CachingHandler) would eliminate cross-sandbox LRU contention entirely — tracked in #3548 as a longer-term improvement.