[ac-pplx]

What

Phase 3 of containing the Direct-vs-Kubernetes split. Shared code that decides something from one row's managed_by (a sandbox's public.sandboxes.managed_by or a node's nodes.managed_by) now asks the shared SubstrateKind a named question instead of matching the variant.

SubstrateKind (pub type SubstrateKind = NodeManagedBy; in crates/libs/common/core/src/types/node.rs) gains one const fn per question. Each doc states only the substrate fact and both variants' answers. What a caller does with the answer, and with a label that does not parse, is in a comment at the call site.

method	substrate fact	Kubernetes	Space	site
proves_execution_absence	Does the substrate durably publish activation ownership before it launches a runtime?	yes	no	sandbox store AutoLifecycleExecution::proves_no_execution
wakes_after_failed_control_connect	When the control path cannot connect, can a wake give the sandbox a runtime? (Direct wakes dormant rows on the access path instead.)	yes	no	api spaced.rs control_runtime_client
listed_in_live_node_inventory	Is a node listed in the live (Kubernetes) node inventory?	yes	no	trigger queue fetch_trigger_targets
restores_through_pods	Does a sandbox restore through a Kubernetes restore pod?	yes	no	cluster controller orphaned_restore_pod_gc/store.rs::row_lifecycle
has_node_drain_reconciler	Does the node-drain reconciler manage the node?	yes	no	api admin/nodes.rs::reconciler_reasserts_cordon

Why methods on the shared type: each question is about the row, whose substrate need not be the running one. The api and cluster-controller sites could reach SubstrateCapabilities (#368), but that would answer for the running substrate instead. The sandbox store and trigger queue can't depend on the api crate at all. Facts about the running substrate stay in SubstrateCapabilities.

Two scopes are now named once:

• Admin node registry (routes/admin/node_maintenance.rs): const REGISTRY_SUBSTRATE: SubstrateKind = SubstrateKind::Space; is used by every registry statement's bind and by update_node's verify guard.
• Node-drain reconciler (cluster/controller/server/src/node_drain/store.rs): DRAIN_RECONCILER_SUBSTRATE is bound by the draining and terminating listings. A unit test pins that it is the only substrate whose has_node_drain_reconciler() is true; an exhaustive match in the test makes a new variant fail to compile until the sweep includes it. has_node_drain_reconciler and SubstrateCapabilities::supports_node_drain link to each other.

docs/rust/space-conventions.md and crates/libs/api/src/controllers/substrate/AGENTS.md record the convention: a decision about a row's substrate calls a SubstrateKind method, a fact about the running substrate reads capabilities(), and code never compares the label string. The impl SubstrateKind doc says the same. Two exceptions: Rust that repeats a statement's fixed scope compares against the named const the statement binds (REGISTRY_SUBSTRATE), and mapping a SubstrateKind variant to a value (wire labels, route identity, construction) may match it. Since #607 made Substrate's variants private, the AGENTS.md bullet follows #607's bullets on them, as_kubernetes() and capabilities(), and names SubstrateKind explicitly.

Behavior

No change for any input. Kubernetes answers each question the way the old == Kubernetes test did, and Space the way the old == Space test did. Sites that read managed_by as text still parse it first, and an unparseable label takes the same branch as before (parse().is_ok_and(SubstrateKind::method) replaces matches!(parse(), Ok(SubstrateKind::Kubernetes))). Typed sites still fail at decode, as before. No SQL text, wire format, schema, metric or configuration changes.

Site table

Every site outside the api crate's substrate module that matches SubstrateKind/NodeManagedBy variants or compares the parsed value, in the crates this phase covers. sbx-mgr, net-gateway and node-api-sim are out of scope.

site	what it does	decision
sandbox/store/src/lib.rs AutoLifecycleExecution::proves_no_execution	a Kubernetes row with no execution evidence has no execution	method proves_execution_absence
cluster/trigger-queue/src/lib.rs fetch_trigger_targets partition	a Kubernetes node missing from the warm live-node set is permanently unavailable	method listed_in_live_node_inventory; fetch_targets_requires_live_direct_row_for_space_nodes now also runs with a populated live inventory and expects the live Direct worker to stay triggerable
cluster/controller/server/src/orphaned_restore_pod_gc/store.rs row_lifecycle	a non-Kubernetes or unparseable row is Unrecognized and skipped	method restores_through_pods
api/src/controllers/sandbox/spaced.rs control_runtime_client	a failed control connect on a Kubernetes row triggers a dormant wake and reconnect	method wakes_after_failed_control_connect
api/src/routes/admin/nodes.rs reconciler_reasserts_cordon	on a Kubernetes node row, the drain reconciler will re-assert cordon	method has_node_drain_reconciler, tied to the reconciler's DRAIN_RECONCILER_SUBSTRATE
api/src/routes/admin/node_maintenance.rs update_node (verify)	only a live Space node may be verified	registry const REGISTRY_SUBSTRATE: the same rule the registry's binds enforce, so it is spelled once
cluster/controller/server/src/transient_lifecycle_sweeper/store.rs failed_active_execution_scope	parses the running substrate's managed_by() label	leave: a fact about the running substrate, which belongs in SubstrateCapabilities; no existing field fits (#368 follow-up)
transient_lifecycle_sweeper/store.rs (managed_by().parse::<NodeManagedBy>())	parses the running substrate's label into a value	leave: value parse
cluster/controller/server/src/orphaned_sandboxes/quarantine_recovery.rs NodeApiTarget { managed_by: Kubernetes }	builds a value	leave: construction
common/core/src/types/sandbox.rs SandboxRouteIdentity::try_new	Kubernetes ⇔ pod UID present	leave: route-identity invariant
common/core/src/types/sandbox.rs effective_substrate_kind_for	a pod UID implies Kubernetes for legacy identities	leave: route-identity inference
common/svc/src/space_api_route.rs claims-vs-local substrate_kind	compares two identities	leave: route identity
internal-api data_plane_sessions/{session_route,service_route,route_token}.rs, route_identity.rs, sandbox_service_ports.rs	parse node_managed_by into a route identity's substrate_kind and pass it through	leave: route identity / wire mapping
internal-api data_plane_sessions/route_rows.rs missing_backend_endpoint_message	prints managed_by	leave: diagnostic text
internal-api lib.rs, published_state.rs, status.rs, node_inventory.rs, template_releases_store.rs; trigger queue; cluster-controller stores; api public_template_operations.rs, admin/templates.rs, public/sandboxes.rs	.bind(SubstrateKind::…)	leave: SQL (out of scope; phase 2 sibling)
api/src/routes/session_auth.rs ConsoleSubstrate::from_managed_by	maps managed_by to the console /me label	leave: wire mapping
api/src/controllers/sandbox/lifecycle.rs delete_observed_sandbox	defaults an unknown node's managed_by to Space when building a NodeApiTarget	leave: a default value; the substrate's endpoint resolution makes the decision

internal-api has no Rust-side substrate decision left after #383; all of its uses are binds or route-identity plumbing.

Validation

check	result
after rebasing onto #603/#607 (b3da5e2): cargo fmt --all -- --check	pass
cargo clippy --workspace --locked --all-targets --all-features -- -D warnings	pass
cargo arch, plus main's architecture-baseline.toml/ARCHITECTURE.toml as base (as CI passes them)	0 required, including #607's kubernetes_access and capability_identity ratchets
pre-commit (gitleaks), pre-commit run --from-ref origin/main --to-ref HEAD	pass
cargo nextest run --all-features --retries 0 -p common-core, sandbox-store, cluster-trigger-queue, api, cluster-controller-server, SPACE_CI_REQUIRE_TEST_DEPS=1, ×3	3312/3312 passed in each run; the 1 skip is the #[ignore] benchmark benchmark_single_node_occupancy
before the rebase: cargo fmt --all -- --check, scripts/check-rust-error-policy.sh	pass
cargo clippy --workspace --locked --all-targets --all-features -- -D warnings	pass
pre-commit (gitleaks), pre-commit run --from-ref origin/main --to-ref HEAD	pass
cargo nextest run -p pplx-space-cluster-trigger-queue -p pplx-space-cluster-controller-server, same flags, on the final change	1020/1020 passed, 0 skipped
cargo nextest run with -p common-core, sandbox-store, cluster-trigger-queue, cluster-controller-server, api, internal-api-server, cluster-reconciler, --all-targets --all-features --run-ignored all --retries 0, SPACE_CI_REQUIRE_TEST_DEPS=1, ×3 on 3930ee1 (production code unchanged since)	3624/3624 passed, 0 skipped, in each run
characterization tests #273/#290/#326/#330	inside the runs above, unchanged, pass
Kind, earlier revision (c73171f, the same production behavior), fresh cluster	smoke-kind-e2e.sh pass; smoke-kind-typed.sh all with CI's env: ECW 79/79, native 24/24, none quarantined. Not rerun, because the review fixes don't change production behavior.

Mutations: each flips one answer (the Kubernetes rows on 3930ee1, the Space rows on the final change), and each is caught by a site or scope test:

mutation	failing test(s)
update_node verify guard compares against Kubernetes	api node_registry_writes_only_direct_nodes (the Kubernetes twin's verify update dials a closed port and gets a gateway error instead of 404)
DRAIN_RECONCILER_SUBSTRATE = Space	cluster-controller drain_listings_bind_the_only_substrate_with_a_drain_reconciler
has_node_drain_reconciler: Kubernetes ⇒ false	that test, plus api only_draining_and_terminating_promise_an_automatic_cordon_retry, the_two_recovery_hints_disagree_exactly_where_they_should, a_cordon_on_a_draining_node_promises_the_retry_the_reconciler_performs
wakes_after_failed_control_connect: Kubernetes ⇒ false	api control_runtime_client_wakes_only_kubernetes_rows_after_an_unreachable_runtime
proves_execution_absence: Kubernetes ⇒ false	sandbox-store proves_no_execution_only_for_kubernetes_rows; api timeout_route_distinguishes_dormant_intent_from_unreachable_execution, dormant_timeout_persists_without_waking_and_caches_retries
listed_in_live_node_inventory: Kubernetes ⇒ false	cluster-reconciler fanout_resolves_live_nodes_once_and_dead_letters_ghosts_without_resolving
restores_through_pods: Kubernetes ⇒ false	cluster-controller classify_sandboxes_reads_lifecycle_and_generation
listed_in_live_node_inventory: Space ⇒ true	trigger-queue fetch_targets_requires_live_direct_row_for_space_nodes (the populated-inventory case)
proves_execution_absence: Space ⇒ true	sandbox-store proves_no_execution_only_for_kubernetes_rows; api timeout_route_distinguishes_dormant_intent_from_unreachable_execution
wakes_after_failed_control_connect: Space ⇒ true	api control_runtime_client_wakes_only_kubernetes_rows_after_an_unreachable_runtime
restores_through_pods: Space ⇒ true	cluster-controller classify_sandboxes_reads_lifecycle_and_generation
has_node_drain_reconciler: Space ⇒ true	cluster-controller drain_listings_bind_the_only_substrate_with_a_drain_reconciler; api only_draining_and_terminating_promise_an_automatic_cordon_retry

🤖 Generated with Claude Code

[the-gigi-pplx]

Review — 2 findings

Behavior is unchanged at every converted site. SubstrateKind has exactly two variants (Space, Kubernetes, crates/libs/common/core/src/types/node.rs:684-687), and all five new methods answer Kubernetes ⇒ true, Space ⇒ false, the same as the Ok(Kubernetes) / == Kubernetes tests they replace in spaced.rs, orphaned_restore_pod_gc/store.rs, trigger-queue/src/lib.rs, sandbox/store/src/lib.rs and admin/nodes.rs. An unparseable label still takes the old branch, since parse().is_ok_and(..) is false on Err. REGISTRY_SUBSTRATE and DRAIN_RECONCILER_SUBSTRATE bind the same values the literals did. I grepped the covered crates (api, cluster, sandbox, internal-api) for remaining variant comparisons, and the only hits are the "leave" rows in the site table (transient_lifecycle_sweeper/store.rs:155, session_auth.rs:33, lifecycle.rs:1239, quarantine_recovery.rs:113). Both findings are about doc prose.

Low

1. crates/libs/common/core/src/types/node.rs:805-809 — The Space bullet says Direct wakes "a dormant sandbox" on the access path, but the Kubernetes bullet just above defines dormant as "suspended, or created and never started".
   DirectSubstrate::access_runtime_client (crates/libs/api/src/controllers/substrate/direct.rs:4199-4202) wakes only Suspended | Restoring rows and connects to everything else directly. Narrow it to "a suspended or restoring sandbox", or say why Direct never has a created-but-unstarted row, so the two bullets don't use one word with two meanings.

2. crates/libs/api/src/routes/admin/node_maintenance.rs:192-194 — "Every registry statement scopes to it, and an update's verify reads only a node of it" overstates it.
   fetch_node (line 233) and fetch_nodes (line 258) read nodes of every substrate. verify reads any node through fetch_node and then rejects a non-Space one at line 559. What's true is that every registry write binds REGISTRY_SUBSTRATE (lines 452, 519, 657, 704) and verify proceeds only for a live node of it. Saying exactly that keeps the "never touches a node another substrate manages" conclusion accurate.

Verified against b3da5e2278c6db3f630f187cc88f0294ca7d634d. I checked behavior preservation by enumerating both variants at every changed site and by grepping for leftover variant comparisons. CI is green on this head. I didn't rerun the test suite or the mutation table in the description, and I didn't run a Kind smoke.

[the-gigi-pplx]

Approved. 2 non-blocking findings in the review comment above.