atelet: poll ateom workload stats into template-level metrics

[Tim Bai (baizhenyu)]

Second piece of #896 (Phase 1 of #550): atelet polls every local ateom's GetActiveWorkloadStats and turns the samples into template-level metrics — the TSDB half of #174's cardinality split. After this PR, "how much CPU and memory is this template using" is a Cloud Monitoring query.

The reader

A statsPoller in atelet, driven by --actor-stats-poll-interval (default 1m; 0 disables; nonzero values are clamped to a 50s floor, the worst-case duration of sampling one micro-VM ateom — 25 containers × 2s per guest-agent call).

• Stateless discovery: each tick lists ateoms/* (entry names are worker pod UIDs — the same sockets the lifecycle RPCs dial) and probes each with the parameterless discovery read. No worker-to-actor mapping, no control-plane dependency, nothing to recover after an atelet restart. Attribution comes solely from the identity echoed in each sample, per the RPC's contract.
• One tolerance rule: any dial or call failure means "not a target this tick" — covering stale directories of deleted workers, ateoms that have not started listening, and teardowns mid-poll. NO_WORKLOAD / NOT_MEASURABLE_YET are skips by the RPC's own contract.
• Bounded concurrency: distinct ateoms are probed 8 at a time (one probe per guest, so nothing the interval floor defends against is reintroduced); a node of stuck-but-accepting sockets degrades to ceil(n/8) timeouts instead of n sequential ones.
• WorkerPool enrichment: one field+label-selected pod LIST per tick maps pod UID → owning pool (ate.dev/worker-pool, the label the pool controller stamps); unresolved pods group without pool labels rather than vanish. Chosen over an informer deliberately — negligible apiserver cost at this cadence, no cache-sync ordering, and the resolver sits behind a function seam if that trade ever changes. Needs one new Downward API env (NODE_NAME); the pods RBAC already existed.

The metrics

Labels on every series: ate.template.namespace/name, ate.sandbox.class, ate.stats.source, ate.workerpool.namespace/name — all bounded sets; actor and atespace identity never reach a metric label (they belong to the events channel, the next PR).

• ate.actor.stats.sampled_actors, …memory_current_bytes, …memory_working_set_bytes — observable gauges over the latest tick's snapshot: each collection observes exactly the groups that currently exist, so a template whose actors leave a node disappears from the export. (Synchronous gauges would re-export their last value until process exit — stale memory for actors long gone.)
• ate.actor.stats.cpu_usage — Float64Counter in seconds (cAdvisor / OTel *.cpu.time convention; the wire stays µs). The raw cpu_usage_usec is cumulative per-epoch per actor, so the poller accumulates per-sweep increases against per-actor baselines: first sight establishes a baseline and charges nothing (atelet cannot tell a new actor from its own restart — re-charging epochs the previous process counted would spike rate()), a decrease is an epoch reset charged from the new value, and baselines are pruned to the actors seen. Bounded imprecision (≤1 interval per actor across restarts; the pre-checkpoint tail) is documented on the instrument; per-actor precision arrives with the lifecycle events.

Validated live on ate-dev

Both source families, simultaneously, with one actor per class:

The same counter application reads 5-6× larger through the cgroup source (whole sandbox: sentry heap, netstack, gofers) than through the guest agent (workload containers only) — the concrete case for the ate.stats.source label and its group-don't-sum rule. Also exercised live: pool labels resolved on every series, restart-without-spike on the CPU counter across a DaemonSet rollout, and OTLP delivery to the gke-managed-otel collector with zero export errors.

Out of scope

• Per-actor events + lifecycle first/final samples — next PR per Actor resource telemetry Phase 1: atelet reader and OTel emission #896.
• k8s.node.name resource attribute (one manifest line, any time).
• e2e coverage — the metrics e2e suite is the natural home once the events channel lands.

Part of #896, toward #550.