nfsproxy: shard NFS handle cache by mount point

[Merlin0220]

Fixes #3555.

Problem

The NFS proxy currently uses a single global LRU file-handle cache shared by all sandboxes and mount points on the same orchestrator node.

Because there is no cache ownership boundary between mounts, filesystem activity in one sandbox can affect the cached NFS handles of another sandbox. For example, listing a directory containing many files creates a large number of handles and can evict directory handles that are still being used by an unrelated sandbox.

When the affected sandbox later reuses an evicted directory handle, the NFS proxy can no longer resolve it and returns:

NFSStatusStale

The client surfaces this error as:

Stale file handle

The most severe case is eviction of the handle representing a sandbox mount point's root directory, which corresponds to the root of its chroot filesystem. This root handle is the entry point through which the NFS client resolves paths within the mount.

If the root handle is evicted, subsequent operations rooted at that handle fail with Stale file handle. The existing mount cannot recreate the root handle through normal filesystem operations because those operations already require a valid parent handle. As a result, the entire mount becomes unusable and cannot recover transparently; the client must remount the filesystem to obtain a new root handle.

Increasing the global cache size only reduces the likelihood of eviction. It does not prevent one sandbox from evicting handles owned by another sandbox.

Fix

Allocate an independent NFS handle cache for every successful mount.

Each mount is assigned a random UUID. NFS file handles are encoded as:

<mount UUID><mount-local file handle>

When resolving or invalidating a handle, the mount UUID selects the correct cache shard, and the remaining bytes are forwarded to that mount's helpers.CachingHandler.

This isolates LRU eviction between mount points. Activity from one sandbox can no longer evict the root handle or other cached handles belonging to another sandbox's mount point, preventing unrelated cache pressure from invalidating the entire mount.

Behavior	Before	After
Handle cache allocation	One cache per NFS proxy	One cache per mount point
Cache capacity	1024 handles shared by all mounts	1024 handles per mount
LRU eviction scope	All sandboxes and mounts	A single mount
Cross-mount cache interference	Possible	Prevented
Mount cache cleanup	Global LRU eviction only	Removed when the sandbox lifecycle releases its network

Changes

• Allocate an independent NFS handle cache for each mount point and use a mount UUID to route file-handle operations to the correct cache.
• Track mount caches by sandbox lifecycle and remove them when the sandbox releases its network.
• Add OpenTelemetry gauges for the number of active cache shards and sandbox lifecycle owners.
• Add tests covering cache isolation and lifecycle cleanup.

Testing

go test ./pkg/nfsproxy/mountcache
go test -race ./pkg/nfsproxy/mountcache
go vet ./pkg/nfsproxy/mountcache

Risk: Memory Usage with Larger Cache Limits

The cache limit now applies to each mount point rather than to the entire NFS proxy. As a result, the maximum number of cached handles is approximately:

cache limit per mount × active sandbox count × average mount points per sandbox

Increasing the per-mount cache limit can therefore significantly increase the orchestrator's total memory usage.

Capacity planning should reserve memory based on the maximum number of sandboxes hosted by a node and the expected number of mount points per sandbox.

The nfs.mount_cache.shards and nfs.mount_cache.owners gauges can be used to observe the active cache footprint and validate those assumptions before increasing the cache size.