perf(api): add per-allocation sandbox cache to eliminate TeamItems MGET bandwidth

[AdaAibaby]

Problem

Closes #3593

TeamItems issues SMEMBERS + batched MGET on every call. With teams of 9 000+ sandbox IDs per team index (observed, per #3571) and multiple API allocations (count_instances in variables.tf), the read bandwidth scales as:

bandwidth ≈ count_instances × team_sandbox_count × avg_payload_size × call_frequency
           ≈ N allocations  × 9 000 keys          × ~1 KB            × 12 calls/min

At N=10 allocations this saturates the API allocation NIC. #3571 (batched MGET) mitigated Redis event-loop stalls but left total bytes transferred unchanged.

Approach

Extend the existing publisher + subscriptionManager pub/sub infrastructure (introduced in #2099 / #2668) to also broadcast sandbox state-change events on globalStorageNotifyChannel. Each allocation maintains a local in-process cache populated by these events. TeamItems reads from the cache after the first cold-fetch, eliminating the O(allocations × team_size) multiplier.

This directly answers the TODO: this should be removed once we have a better way to handle node sync comment left in the deleted memory backend (sync.go, removed in #2750): the pub/sub channel is that better way. Unlike the old memory.Storage (a standalone source of truth per allocation), this is a cache layer — Redis remains the source of truth.

Changes

New files

• sandbox_event.go — sandboxEvent JSON type published alongside existing plain routing-key strings. JSON prefix { is an unambiguous discriminator (sandbox:storage: / lock: routing keys never start with {).
• sandbox_cache.go — sandboxCache with sync.RWMutex, keyed by sandbox ID, indexed by team, with per-team warm/cold tracking.
• sandbox_cache_test.go — unit tests for the cache in isolation (no Redis).

Modified files

• publisher.go — publishSandboxEvent: marshals event to JSON, enqueues on the existing 32-worker pool.
• subscription_manager.go — dispatch now detects JSON event payloads and applies them to the embedded sandboxCache before routing-key fan-out. Fully backward compatible: non-JSON payloads follow the existing path unchanged.
• operations.go — Add/Update/Remove broadcast events after each successful Redis write. TeamItems checks the cache (warm path, zero Redis reads) or falls back to SMEMBERS+MGET and warms the team (cold path, once per allocation lifetime per team). Gated by SandboxTeamItemsCacheFlag (default false).
• main.go — cacheForced bool field for test-time override.
• featureflags/flags.go — SandboxTeamItemsCacheFlag = NewBoolFlag("sandbox-team-items-cache", false).
• team_items_test.go — 9 new integration tests (real Redis via testcontainers).

Consistency model

The cache is eventually consistent with Redis. Dropped pub/sub events (backpressure, pod restart) cause temporary staleness until the next cold-fetch for that team. Callers that require strong consistency (ExpiredItems, Reconcile) are unchanged and bypass the cache entirely.

Expected impact

At 100 sandbox mutations/second, 10 allocations:

	Bandwidth
Before (#3571)	~1 080 MB/min
After (cache warm)	~60 MB/min — 18× reduction

At steady state, TeamItems requires zero Redis reads per call.

Rollout

The cache is off by default (sandbox-team-items-cache = false). Flip the LaunchDarkly flag to true in staging first, then production. No config changes required.

Test plan

• Unit: sandbox_cache_test.go — apply/evict/warmTeam/getTeam, state filter, team isolation, stale-entry eviction, event marshal/unmarshal, routing-key disambiguation (10 tests, no Docker required)
• Integration: team_items_test.go — cold-start warming, empty-team warm, event-driven add/remove reflection, end-to-end Add/Remove/Update broadcast via real pub/sub, team isolation, flag-off behaviour (15 tests, testcontainers Redis)
• Regression: full ./internal/sandbox/storage/redis/... suite passes (go test -count=1 -timeout 300s)