Control plane: cross-request OrgKelSnapshot cache (tip-SAID keyed), gated on a written fleet-scale target

[bordumb]

Context

The N+1 KEL replay in the list endpoints is fixed: list_agents / list_fleet now load one shared OrgKelSnapshot per request instead of one per member, and the invariant is pinned by a counting test double on RegistryBackend::visit_events (crates/auths-api/tests/cases/kel_replay_count.rs — full org-KEL replays must be member-count-independent and within a small constant budget, currently 3: the list() roster scan plus OrgKelSnapshot::load's roster + event walk).

What remains is cross-request cost: every request still replays the org KEL from Git 3 times. Cost per request is O(KEL events), and KEL events grow with member churn (~2-3 events per add/revoke/rotate). Rough shape:

Org scale	KEL events (approx)	Event reads per list_fleet request
100 agents	~300	~900
1,000 agents	~3,000	~9,000
10,000 agents + churn	~30,000	~90,000

Decision needed first (do not skip)

Write down the launch-scale assumption (max agents per org, max KEL events) in docs/architecture/. This issue's work is gated on a real org approaching that number — building the cache speculatively adds a correctness surface for no measured benefit.

Security constraint (non-negotiable)

Authority is decided by KEL replay, fail-closed. Any cache of KEL-derived state MUST be invalidated by KEL tip SAID (content-addressed), never by TTL. A TTL'd entry is a revocation-latency hole: a revoked agent would stay authorized until expiry. Cache key = (prefix, tip.said); every request does one cheap get_tip(); SAID mismatch → rebuild. This makes the cache correctness-neutral by construction.

Implementation ladder

Step 1 — in-process, no infra (~40 lines). The control plane serves a single org (AppState::ensure_org pins one org_prefix), so the cache is one entry:

// on AppState
snapshot_cache: Arc<RwLock<Option<(Said /* tip */, Arc<OrgKelSnapshot>)>>>

Per request: get_tip() → if SAID matches, reuse the snapshot; else rebuild and swap. Hot-path cost drops from 3 full replays to 1 tip read. Update the budget constant + comment in kel_replay_count.rs deliberately (the test is designed to make this change visible).

Step 2 — multi-process/multi-tenant only. Port the Tier 0 plumbing from the archived _archived/auths-cloud/crates/auths-cache/src/redis_cache.rs (bb8 pool, TierZeroCache trait shape; the current RegistryBackend::write_key_state docs still describe this Redis-Tier-0/Git-Tier-1 write-through). Two required changes when porting:

• Replace set_ex TTL expiry with tip-SAID keying (see constraint above).
• Cache OrgKelSnapshot (roster + events), not KeyState — the fleet replays are raw event walks, so KeyState caching removes none of them.

Explicitly do not import the archived stack's "Redis as source of truth" principle — that was the deleted bearer-token model; here the KEL is the source of truth and the cache is strictly subordinate.

Acceptance

• Launch-scale assumption documented (agents/org, KEL events) with this issue linked
• Step 1 only: single-entry tip-keyed snapshot cache on AppState; kel_replay_count.rs budget updated to 1 tip read + ≤1 full replay (cold) and still member-count-independent
• A revocation test: revoke an agent, immediately list — the revoked agent reports revoked: true on the very next request (no stale window)
• Step 2 tracked separately if/when multi-process serving lands

References

• crates/auths-api/src/control_plane.rs (list_agents, list_fleet)
• crates/auths-sdk/src/domains/org/delegation.rs (OrgKelSnapshot, OrgSnapshotCache)
• crates/auths-api/tests/cases/kel_replay_count.rs (regression guard + budget constant)
• docs/plans/audit.md — finding D1, done-signal 5, Open Question 5
• _archived/auths-cloud/crates/auths-cache/ (salvage source for Step 2)

[bordumb]

Scale target (the gating acceptance item) — now written down

Target: 10,000+ agents per org (1,000 engineers × ~10 agents each), implying ~30–50k org-KEL events after a year of add/rotate/revoke churn, with sustained churn (the tip moves constantly).

What the number changes in the design above

Step 1 as originally written (rebuild-on-invalidate) is necessary but not sufficient at this scale. Cache hit rate collapses exactly when the org is busiest: every add/revoke/rotate moves the tip, and each invalidation pays a full O(50k-event) rebuild (~3 full replays ≈ 150k Git event reads ≈ seconds per request). Pagination does not help — list() and OrgKelSnapshot::load scan the full KEL regardless of page size, so a 10k-agent fleet view at 200/page is ~50 requests × full replays.

Step 1 upgrades to incremental advance. KELs are append-only and sequentially numbered, so a snapshot valid at tip seq N is advanced to tip M by folding only events N+1..M via visit_events(prefix, N+1, ...) — same replay semantics, resumed instead of restarted. Steady-state per-request cost becomes O(churn delta) regardless of KEL size. The tip-SAID check stays as the correctness gate (no stale-authority window; fail-closed survives intact). Memory: a 50k-event snapshot is ~50–100MB, acceptable for a single-org control plane.

crates/auths-api/tests/cases/kel_replay_count.rs remains the regression floor either way; its budget gets updated deliberately when this lands.

The bulk-onboarding write-path ceiling this scale also exposes is a separate problem — tracked in its own issue (see reference below).

[bordumb]

Scale target decided (this unblocks the gate)

Sales target: engineering orgs of 1,000+ engineers at ~10 agents each → 10,000+ agents in a single org.

Derived planning numbers:

• Org KEL: ~2-3 org-KEL events per agent add (anchoring ixn + scope seal ixn) + rotation/revocation churn → ~30k-50k events within year one
• Today's cost: 3 full replays × 50k events ≈ 150k Git event reads (~7-15s) per list request, and pagination does not help — list() + OrgKelSnapshot::load scan the full KEL regardless of page size, so a 10k-agent fleet view at 200/page ≈ 7.5M event reads
• Churn is sustained at this org size, so the tip moves constantly

Design upgrade: Step 1 is now incremental advance, not rebuild-on-invalidate

The original Step 1 ("tip SAID mismatch → rebuild") collapses under sustained churn: every add/revoke/rotate moves the tip, so the cache invalidates exactly when the org is busiest, and each invalidation pays the full O(50k) rebuild.

KELs are append-only and sequentially numbered, so a snapshot valid at tip seq N advances to tip M by folding in only events N+1..M (visit_events(prefix, N+1, ...)) — same replay semantics, resumed instead of restarted. Steady-state cost per request becomes O(churn delta) regardless of KEL size. The content-addressed tip SAID still guards correctness: fail-closed survives intact, no stale-authority window.

Requires OrgKelSnapshot to support advance(&mut self, new_events) — extending the roster, scope seals, and revocation set incrementally. Full rebuild remains only the cold-start path.

Memory check: a 50k-event snapshot is ~50-100MB — acceptable for the single-org control plane.

Revised acceptance

• Launch-scale assumption documented: 10k+ agents / ~50k org-KEL events / sustained churn (this comment; mirror into docs/architecture/ when implementing)
• OrgKelSnapshot::advance — incremental fold from cached seq to new tip
• Single-entry tip-keyed cache on AppState: get_tip() per request; SAID match → reuse; SAID mismatch → advance (never rebuild unless cold)
• kel_replay_count.rs budget updated: steady-state = 1 tip read + O(delta) event reads; still member-count-independent
• Revocation test: revoke → immediate list reports revoked: true (no stale window)

The write-side ceiling this scale exposes (bulk-onboarding 10k agents onto one sequential kt=1 KEL) is tracked separately: see the linked issue.

[bordumb]

Gate noted in the v0.1.3 audit train: the org-KEL N+1 elimination (milestone 2, commit f4bc3e8 lineage, released in 0.1.3) satisfies the prerequisite this issue was waiting on. Cache design can proceed against the 0.1.3 control-plane shape.