Prefer workers with cached actor images during scheduling

[han2ni3bal-pixel]

Description

Atelet maintains a node-local image cache, but the scheduler does not know which images are cached on each node. As a result, an actor may be assigned to a cold node and pull its images again, even when another eligible node already has them cached.

Image preparation happens synchronously during ResumeActor, so unnecessary pulls increase actor startup latency and registry traffic.

Proposal

• Let atelet report its cached image digests to the control plane.
• Store cache information by node.
• After applying existing constraints, prefer available workers whose node has all required actor images cached.
• Fall back to the current random selection when no cache hit is available.

Local snapshot availability remains a hard constraint. Image cache affinity is applied only among nodes that can restore the selected snapshot.

A general scoring framework is not required: use cache-hit preference with fallback.

Expected Outcome

Reduce unnecessary image pulls and improve Actor Resume latency and consistency.

Related Issues

• Avoid re-untarring actor image rootfs on every restore #166 and Use a node-local overlayfs rootfs cache to eliminate per-restore untar during actor resume #228 optimize node-local image/rootfs reuse.
• RFC: Flexible & Non-Ossifying Control Plane Persistence Layer #135 discusses locality-aware scheduling for cached snapshots.
• Allow large image pulls to succeed by decoupling workflow deadline from lock TTL #233 tracks cold image pulls exceeding the ResumeActor deadline.

This proposal complements those efforts by making worker selection aware of node-local image cache availability.

[han2ni3bal-pixel]

Following up on Julian Gutierrez Oschmann (@juli4n)'s comment on PR #298 — I agree we should agree on a unified approach for node-locality before landing more one-off scheduling features.

Where we are today

Only one node-locality mechanism is merged: when an actor is paused with a local snapshot, NodeVmsWithLocalSnapshots on the Actor record is a hard constraint during ResumeActor — workers must be on one of those nodes.

#276 / PR #298 proposes a second path (not merged yet): atelet periodically reports OCI image digests, control plane stores them per node with a TTL, and the scheduler prefers workers whose node already has the required images, falling back to random selection.

The concern isn't that we already run two models in production — it's that merging #298 as a standalone RPC and store key sets a pattern that #228 (rootfs overlay), #135 (hot snapshot cache), and future volume locality will likely copy unless we align now.

Proposed direction

Split two concerns:

1. Actor-bound, durable, hard constraints — e.g. pause local snapshot nodes. Stays on the Actor record.
2. Node-bound, TTL-backed, soft preferences — e.g. warm images, extracted rootfs, resident snapshots. One shared node inventory reported periodically by atelet and stored per node. Stale or wrong data should only hurt latency, not correctness.
   Keep the Prefer workers with cached actor images during scheduling #276 scheduling model: fixed layers (hard filter → ordered soft preferences → random fallback), not a general scoring framework.

Soft preference order (when we add them): hot snapshot on node first, then rootfs overlay (#228), then image tar cache (#276). Pause local snapshots remain hard-only via the Actor record — don't move them into node inventory.

Rough plan

Phase 0 (this issue): agree on hard vs soft split, node inventory as the long-term envelope, and preference ordering. Don't ossify ReportNodeImageCache as the permanent API.
Phase 1: implement #276 on NodeInventory directly (rebase PR #298 onto that, rather than merge then rename).
Phase 2: add rootfs overlay digests from #228.
Phase 3: add resident snapshot prefixes for Suspend hot-restore (#135 light).
Phase 4 (later): secondary indexes / #135 ListOptions only if query patterns prove we need them.
Happy to help turn Phase 0 into a short design note here if that's useful.