[nickdujay-pplx]

Problem

A rolling upgrade must keep template lookups working when client and server versions differ. The scope protocol reverted in #734 required an acknowledgement from servers that did not implement it. That can stop sandbox creation and restore during an otherwise normal upgrade.

This is the compatibility stage for replacing #217. It does not restore its stricter authorization rules.

Evidence

The incompatible pairing is a new lookup client with an old template-manager. The old server returns a valid template response without the new metadata. Rejecting that response makes successful lookups fail solely because of version skew. The reverse pairing must also retain existing cache and offline behavior.

Design

Clients send scope metadata through both cached and cold lookups. On-demand catalog resolution preserves the original scope, including legacy requests without scope; background and prefetch pulls use cluster scope. Servers preserve existing lookup and authorization decisions and return a separate receipt. Missing receipts retain legacy behavior; mismatched receipts fail. RPC errors are not retried without scope.

A receipt proves only that one peer received metadata. It is deliberately different from an authorization acknowledgement, so later enforcement cannot mistake this preparatory version for an enforcing server. No schema or placement change is included.

Validation

• Eight cold-cache regression cases fail before the review fix because the catalog receives cluster scope. All ten cases pass afterward, covering exact digests and name-only requests with local, node, sandbox, cluster, or absent scope. These use real template-manager Unix-socket requests and a real catalog/PostgreSQL fixture.
• Existing background/prefetch cases now assert the exact downstream cluster scope. All 34 focused cases and all 270 non-privileged template-manager tests pass. The two unchanged privileged filesystem cases were not rerun for this narrow fix. Formatting and all-target/all-feature Clippy pass for the library and service binary.
• Real old/new node-api and template-manager binaries pass 64 component scenarios across all four pairings and catalog fixtures with/without receipt support. Cold requests preserve the expected downstream scope; warm requests succeed without catalog access; unauthenticated requests are rejected. Node-api runs under its service UID and documented socket group.
• The cold fixture returns a release with no storage locations: the replay proves catalog resolution and scope transport, not a completed download or VM create/restore/build. Catalog receipt behavior and unrelated startup probes are synthetic in the binary replay. The separate Rust RPC regression uses the real catalog.
• Before this revision, full hosted CI, including all four Kind/Firecracker shards, passed on 0fb8c4be8. Fresh CI is required for the review fix.

Earlier validation also demonstrated all four pre-protocol/compatible binary pairings, cached offline behavior, and red/green create/restore request regressions when the strict missing-response rule was restored. It did not prove a completed mixed-version VM lifecycle.

Rollout

Deploy compatibility first. Inspect auth.template_lookup.compatibility across participating services and exercise catalog lookups as well as cache hits. Zero legacy observations on an idle service is not proof of readiness. Strict authorization remains a separate follow-up with its own rollout and tests.

Severity: availability regression during rolling upgrades. This preparatory PR does not claim to complete authorization hardening.

AI assistance: OpenAI Codex contributed the implementation, tests, and PR description.

[shrikant-pplx]

Re-reviewed 0b214e201 after the requested functional change.

1. Root cause fixed: Yes.
2. Every affected path covered: Yes.
3. Changes remain focused: Yes.
4. Tests reproduce the failure and cover important edges: Yes; CI is green.
5. Introduced P0/P1 regression: No.

No P0/P1 findings remain.