[ac-pplx]

Phase 2 of containing the Direct/Kubernetes substrate split, last PR: the same method as #373 (cluster controller), #383 (internal-api, trigger queue, sandbox store) and #506 (api crate). Based on main (9af70ec). One commit.

What merging does

Control-plane Postgres sandboxes.managed_by and nodes.managed_by hold the substrate that owns a row, 'k8s' or 'space'. After #373, #383 and #506, these production sites still spelled it:

1. the shared routable_execution_or_non_k8s fragment (crates/libs/common/svc/src/sql/sandbox_routability.rs), which wrote {alias}.managed_by <> 'k8s' into four statements in three crates;
2. the Kubernetes adapter's builder lookup (n.managed_by = 'k8s');
3. the Direct route-identity match ("space" => / "k8s" =>);
4. Direct's create-target claim and release (managed_by = 'space'), which Preserve authenticated scope in cached template lookups #217 added while this PR was open.

They now bind SubstrateKind or match its parsed variant. A new archcheck instance, pattern_ratchet.substrate_literals, catches the common forms of a new literal in production code. no_crossing, listed_no_growth and stale_entries are required. no_new_entries and no_raised_caps are advisory, so a deliberate exception goes in as a reviewed baseline entry, as in the sibling instances.

There are no schema, wire format, metric or configuration changes, and no behavior change. SubstrateKind encodes as the column's text, so a bound SubstrateKind::Kubernetes matches exactly what 'k8s' did.

Converted sites

Each new placeholder is numbered after the statement's existing parameters, and its .bind comes last.

#	File	Statement	Before → after
L1	cluster/controller/server/src/transient_lifecycle_sweeper/store.rs	list_aged_transient_query (routable_execution select-list fact)	fragment <> 'k8s' → routable_execution_or_non_k8s("sandboxes", "$5"), $5 = Kubernetes
C1	same	converge_restoring_query (converge CAS)	→ "$5", $5 = Kubernetes
P1	internal/api/server/src/published_state.rs	publish_sandbox_states restored-state convergence	→ routable_execution_or_non_k8s("s", "$4"), $4 = Kubernetes
W1	api/src/controllers/sandbox/wake_completion.rs	fenced_completion_cas (cancelled-wake redrive)	→ "$2", $2 = Kubernetes
K1	api/src/controllers/substrate/kubernetes/adapter/substrate_adapter.rs	builder_client_for_node lookup	n.managed_by = 'k8s' → $2 = Kubernetes
D13	api/src/controllers/substrate/direct.rs	claim_create_target (#217)	managed_by = 'space' → $3 = Space
D14	same	release_create_target (#217)	managed_by = 'space' → $3 = Space
R1	api/src/controllers/substrate/direct.rs	load_active_direct_route_identity's substrate_kind match	match substrate_kind.as_str() { "space" => Direct, "k8s" => Kubernetes, other => Err } → route_identity_substrate_kind(&str) -> Option<SandboxSubstrateKind>, which parses SubstrateKind and matches the variant

• The fragment takes the placeholder as a second argument, routable_execution_or_non_k8s(alias, kubernetes_placeholder), as cluster-controller: bind the substrate value instead of writing it in SQL #373 did for row_state_guards. Every caller is listed above (rg routable_execution_or_non_k8s finds no others). Its doc comment, which explained why the literal was left as debt, now says what the argument is. The fragment stays a fragment. Inlining it into four statements would give the routability contract four copies, which is what the shared definition exists to prevent.
• R1 behavior is identical for every input. NodeManagedBy::parse is an exact match on "space" and "k8s", with no trimming or case folding. Any other text, including "SPACE", " k8s" and "", still returns the same SubstrateError::Failed with the stored text in the message. {substrate_kind:?} formats the String exactly as {other:?} formatted the &str. The mapping moved into a small free function so a unit table can pin every input. Postgres can't produce the 'k8s' or unknown arms, because sandbox_direct_route_identities_substrate_kind_check limits the column to 'space'.
• Enforce tenant publication grants after preparation rollout #458's publisher lookup. While this PR was open, Enforce tenant publication grants after preparation rollout #458 added publisher_node_for_identity (template_releases_store.rs, n.managed_by = 'k8s'). This PR converted it, with a relabelled-builder case in tenant_publication_rejects_invalid_grants that failed its predicate-removal mutation. Revert "Enforce tenant publication grants after preparation rollout (#458)" #606 then reverted Enforce tenant publication grants after preparation rollout #458, so the rebase drops both: those two files are identical to main.
• The Kubernetes adapter's managed_by() label ("k8s") and Direct's ("space") are labels the adapters return, not comparisons; the rule doesn't match them.

Not converted, by design

• crates/libs/sbx/mgr/server/src/route_activation_claims.rs substrate_kind = 'k8s': node-local sbx-mgr SQLite, not control-plane managed_by (sbx-mgr's NodeManagedBy/route substrate is a separate, node-side notion). The rule's SQL patterns name managed_by only.
• crates/libs/node/mgr/server/src/slot_pool/liveness.rs component == "space": a component name, not a substrate value.
• Test files, #[cfg(test)] support modules and fixtures keep writing literal rows (see the rule's allow).

The rule: [rules.pattern_ratchet.substrate_literals]

ARCHITECTURE.toml only, as the archcheck README's "Adding an instance" describes. The patterns, by group:

patterns = [
  # SQL
  '''(?i)\b\w*managed_by\b"?\s*(?:::\s*text\s*)?\)?\s*(?:=|<>|!=|\bIS\s+(?:NOT\s+)?DISTINCT\s+FROM\b)\s*'(?:k8s|space)'''',  # S1
  '''(?i)'(?:k8s|space)'\s*(?:::\s*text\s*)?(?:=|<>|!=|\bIS\s+(?:NOT\s+)?DISTINCT\s+FROM\b)\s*(?:\w+\s*\(\s*)?(?:\w+\.)?"?\w*managed_by\b''',  # S2
  '''(?i)\b\w*managed_by\b"?\s*(?:::\s*text\s*)?\)?\s*(?:=\s*ANY|(?:NOT\s+)?IN)\s*\((?:[^()]|\([^()]*\))*?'(?:k8s|space)'''',  # S3
  '''(?i)'(?:k8s|space)'\s*(?:NOT\s+)?IN\s*\([^)]*\bmanaged_by\b''',  # S4
  '''(?i)\bCASE\s+(?:\w+\s*\(\s*)?(?:\w+\.)?"?\w*managed_by\b.*?\bWHEN\s+'(?:k8s|space)'''',  # S5
  # Rust comparisons with the label, a label const or a variant's label
  '''\b\w*(?:managed_by|substrate)\w*\b[\w.()&*]*\s*(?:==|!=)\s*(?:Some\s*\(\s*)?(?:r#*)?"(?:k8s|space)"''',  # R1
  '''"(?:k8s|space)"#*\s*\)?\s*(?:==|!=)\s*[\w.()&*]*\b\w*(?:managed_by|substrate)\w*\b''',  # R2
  '''\b\w*(?:managed_by|substrate)\w*\b[\w.()&*]*\.eq(?:_ignore_ascii_case)?\s*\(\s*(?:r#*)?"(?:k8s|space)"''',  # R3
  '''\bmatches!\s*\(\s*[^,]*\b\w*(?:managed_by|substrate)\w*\b[^,]*,[^;]*"(?:k8s|space)"''',  # R4
  '''(?:==|!=)\s*&?[\w:]*(?:NODE_MANAGED_BY_(?:K8S|SPACE)\b|(?:SubstrateKind|NodeManagedBy)::\w+\s*\.\s*(?:as_str|to_string)\s*\(\s*\))''',  # C1
  '''(?:\bNODE_MANAGED_BY_(?:K8S|SPACE)\b|\b(?:SubstrateKind|NodeManagedBy)::\w+\s*\.\s*(?:as_str|to_string)\s*\(\s*\))\s*(?:==|!=)''',  # C2
  # Rust matches on the label string, and arms on it beside a substrate name
  '''\bmatch\s+[\w.&*]*\b(?:\w*managed_by|substrate_kind)\b(?:\s*\(\s*\))?\s*\.\s*(?:as_str|as_deref|as_ref)\s*\(\s*\)\s*\{''',  # M1
  '''\bmatch\s+[\w.&*]*\bmanaged_by\s*\(\s*\)\s*\{''',  # M2
  '''"(?:k8s|space)"[^;{}=:]*=>.*(?:[Ss]ubstrate|ManagedBy|managed_by)''',  # A1
  '''(?:[Ss]ubstrate|ManagedBy|managed_by).*"(?:k8s|space)"[^;{}=:]*=>''',  # A2
]

(The # S1… labels are only for this table; ARCHITECTURE.toml has the patterns without them.)

• SQL (S1–S5): managed_by compared with 'k8s'/'space' by =, <>, != or IS [NOT] DISTINCT FROM, on either side (S1, S2). The column may have an alias, a prefix (node_managed_by) or quoting (n."managed_by"), a ::text cast or a wrapping call (lower(…)). "Either side" applies to these binary comparisons only. For lists, managed_by [NOT] IN (…) or = ANY(…) with a literal in the list counts (S3; one level of nested call, as in (lower($1), 'k8s')), and so does 'k8s' [NOT] IN (…) naming managed_by (S4). CASE managed_by WHEN '…' counts on one line (S5). SET managed_by = '…' counts too.
• Rust comparisons (R1–R4, C1–C2):
  • R1–R4: an identifier containing managed_by or substrate compared with "k8s"/"space", raw strings included. Forms: ==/!= either side, Some("…"), .eq, .eq_ignore_ascii_case and matches!.
  • C1/C2: any value compared, either side, with a NODE_MANAGED_BY_* const or with a SubstrateKind/NodeManagedBy variant's .as_str()/.to_string(). Those were the likeliest way around a flagged == "k8s"; the consts stay public in this PR.
• Matches (M1, M2, A1, A2):
  • M1: a match on a *managed_by/substrate_kind label string (.as_str()/.as_deref()/.as_ref()). M2: a match on the adapter's managed_by().
  • A1/A2: a match arm on "k8s"/"space" ("k8s" =>, Some("k8s") =>, ("k8s", x) =>, "k8s" | … =>, "k8s" if … =>) only when a substrate name (substrate, Substrate, ManagedBy, managed_by) is on the same line. That keeps the arm patterns off the non-substrate "k8s"/"space" vocabularies: auth mode, the metrics backend tag, the nftables table, the cgroup component, the lease namespace and the runtime class.
• allow: archcheck's test files are filename-based (tests.rs, *_tests.rs, tests/), so #[cfg(test)] support modules count as production. The allow list names them explicitly: **/*_tests/**, **/test_support.rs, **/test_support/**, **/*_test_support.rs, **/test_fixtures.rs, **/*_test_fixture.rs, **/scripted_test_doubles.rs and crates/libs/db-test-fixtures/**. Fixtures are not converted. I checked each *_tests/ directory under src/ (published_state_tests, adapter_characterization_tests, k8s_store_postgres_tests): all are test modules.
• // and /// lines don't count (archcheck's code-line definition).
• Modes: no_crossing, listed_no_growth and stale_entries are required. no_new_entries and no_raised_caps are advisory, like kubernetes_access, capability_identity, rootfs_copy and node_manager_commands: a deliberate exception is a reviewed baseline entry, not a wider allow. There is no baseline table, since production code has no matches.
• Known limits (the instance's comment has the same list):
  • Not caught:
    • a literal in a VALUES list, split across lines, built with format!, or held in a non-.rs file;
    • if let "k8s" = … and let … else patterns;
    • ["space", "k8s"].contains(…);
    • a column read by name (row.get::<String, _>("managed_by") == "k8s");
    • a label under a neutral name (|m| m == "k8s");
    • a case-folded literal (eq_ignore_ascii_case("K8S"));
    • a match written match &*x.managed_by, &x.managed_by[..] or .as_deref().unwrap_or(…), and so its arms;
    • an arm whose substrate name is only on the match line, or with any of ;{}=: between the literal and => (a guard with ==, ("k8s", NodeRole::Builder), a struct pattern).
  • False positives:
    • the same text in a trailing // or /* */ comment, in #[doc = "…"], in a SQL -- comment, or in an ordinary Rust string such as an error message;
    • a non-substrate vocabulary on a line that names a substrate;
    • an .as_ref()/.as_deref() match on a typed Option<SubstrateKind>/Option<NodeManagedBy> named substrate_kind or *managed_by.

What the rule matches, before and after

Tree	substrate_literals result
c1187fb (this task's starting commit) with this ARCHITECTURE.toml	3 no_crossing (required) findings, 5 lines: direct.rs 1026–1028 (the route-identity match substrate_kind.as_str() { header and its two arms), substrate_adapter.rs 1902, sandbox_routability.rs 61. Without the allow list, the test-support module published_state_tests/lifecycle_writers.rs adds 3.
17cd654 (main while #458 was in)	4 findings, 8 lines: the above plus direct.rs 3119/3142 (#217) and template_releases_store.rs 150 (#458)
9af70ec (this PR's base) with this ARCHITECTURE.toml	3 no_crossing (required) findings, 7 lines: direct.rs 5 (1026–1028, 3107, 3130), substrate_adapter.rs 1902, sandbox_routability.rs 61. That is the 6 converted lines plus the header of the converted route-identity match, which the new M1 pattern catches. cargo arch exits 1.
this head	no findings. cargo arch exits 0.

The rule caught #217's and #458's new literals as soon as I rebased, which is the ratchet working. The only 'k8s'/'space' left in production .rs SQL (outside comments, test files and the allowed support modules) is sbx-mgr's node-local substrate_kind = 'k8s'.

Synthetic probe. A scratch production file (crates/libs/common/svc/src/sql/zz_probe.rs, deleted afterwards) held 100 lines: 52 positives, 35 look-alikes, the 11 listed misses and the 2 listed false positives. A script checked every line against every pattern (rg, the same regex engine). Every positive matched its own pattern; no look-alike or listed miss matched anything; both listed false positives matched. archcheck counted exactly 54 lines (52 + 2).

Pattern	Positives (all matched)	Look-alikes (none matched)
S1	n.managed_by = 'k8s'; MANAGED_BY<>'space'; != 'k8s'; IS DISTINCT FROM 'space'; SET managed_by = 'space'; managed_by::text = 'k8s'; lower(managed_by) = 'k8s'; n.node_managed_by = 'k8s'; n."managed_by" = 'k8s'	managed_by = $2; pool.managed_by = 'autoscaler'; role = 'k8s'
S2	'k8s' = s.managed_by; 'space' <> lower(s.managed_by); 'k8s' = n."managed_by"	'k8s' = s.role
S3	IN ('space', 'k8s'); not in ('k8s'); IN (lower($1), 'k8s'); = ANY(ARRAY['k8s']); "managed_by" IN ('k8s')	IN ($2, $3); IN ($2) AND kind = $3; = ANY($1)
S4	'k8s' IN (n.managed_by); 'space' NOT IN (managed_by, other)	'k8s' IN ($1); 'k8s' IN (n.role)
S5	CASE s.managed_by WHEN 'k8s'	CASE s.managed_by WHEN $1; CASE s.role WHEN 'k8s'
R1	row.managed_by == "k8s"; .as_str() != "space"; node_managed_by == "k8s"; .as_deref() == Some("k8s"); s.substrate.managed_by() == "k8s"; substrate_kind == "space"; == r"k8s"; == r#"space"#	component == "space"; mode == "k8s"; mode == r"k8s"; managed_by == SubstrateKind::Kubernetes
R2	"k8s" == managed_by; Some("k8s") == row.managed_by.as_deref(); r#"k8s"# == row.managed_by	"k8s" == mode
R3	managed_by.eq_ignore_ascii_case("k8s"); managed_by.eq("space"); managed_by.eq(r"space")	mode.eq_ignore_ascii_case("k8s")
R4	matches!(row.managed_by.as_str(), "k8s"); matches!(row.managed_by.as_deref(), Some("k8s" | "space"))	matches!(row.managed_by.parse::<SubstrateKind>(), Ok(SubstrateKind::Kubernetes)); matches!(mode, "k8s")
C1	row.managed_by == NODE_MANAGED_BY_K8S; managed_by != types::NODE_MANAGED_BY_SPACE; .as_str() == SubstrateKind::Kubernetes.as_str(); label == NodeManagedBy::Space.to_string()	NODE_MANAGED_BY_K8S => Ok(Self::Kubernetes), (the parser); .bind(SubstrateKind::Kubernetes.as_str()); kind == SubstrateKind::Kubernetes; pub const NODE_MANAGED_BY_K8S: &str = "k8s";
C2	NODE_MANAGED_BY_K8S == row.managed_by; SubstrateKind::Kubernetes.as_str() != managed_by	(as C1)
M1	let substrate_kind = match substrate_kind.as_str() { (main's direct.rs 1026); match row.managed_by.as_deref() {; match row.node_managed_by.as_ref() {	match row.managed_by.parse::<SubstrateKind>() {; match row.managed_by {; match row.managed_by.as_str().parse::<SubstrateKind>() {; match mode.as_str() {; match substrate_outcome.as_ref() {; match self.substrate.as_ref() {; match substrate_capabilities.as_ref() {
M2	match self.substrate.managed_by() {	(as M1)
A1	"space" => proto::SandboxSubstrateKind::Direct,; Some("k8s") => SubstrateKind::Kubernetes,; ("k8s", true) => NodeManagedBy::Kubernetes,; "k8s" | "kubernetes" => Substrate::from_kubernetes(k),; "k8s" if strict => Substrate::from_kubernetes(k),	"k8s" => AuthMode::K8s,; "space" => Table::Space,; "k8s" => 1,
A2	match managed_by { "k8s" => 1, _ => 0 }	NodeManagedBy::Kubernetes => "k8s",; one-line match k { NodeManagedBy::Space => "space", NodeManagedBy::Kubernetes => "k8s" }

• Listed misses (none matched): if let "k8s" = row.managed_by.as_str() {; let "k8s" = row.managed_by.as_str() else { … }; ["space", "k8s"].contains(&row.managed_by.as_str()); row.get::<String, _>("managed_by") == "k8s"; |m: &str| m == "k8s"; row.managed_by.eq_ignore_ascii_case("K8S"); match &*row.managed_by {; match &row.managed_by[..] {; match row.managed_by.as_deref().unwrap_or("space") {; ("k8s", NodeRole::Builder) => SubstrateKind::Kubernetes,; "k8s" if role == "builder" => SubstrateKind::Kubernetes,.
• Listed false positives (both matched): match substrate_kind.as_ref() { on a typed Option<SubstrateKind> (M1), and an error message format!("expected managed_by = 'k8s' for {id}") (S1).

Precision across the whole tree. I ran the patterns over every .rs file, tests included. On this head they match 24 code lines, all in test files (C1/C2 match nothing anywhere, tests included) and all substrate values (fixture SET managed_by = … writes and managed_by == "k8s" branches). The lines they don't match that mention managed_by or substrate_kind with a literal are inserts, binds, assignments and assertions. The exception is sbx-mgr's node-local substrate_kind = 'k8s', which is out of scope as above.

Evidence

A Postgres-backed test per converted statement

Each test shows a row of the statement's substrate on one side of the predicate and an otherwise identical row of the other substrate on the other. For the fragment, the discriminating case is a row without a routable execution: the 'k8s' row must not converge, and the 'space' row must, because only Kubernetes has the routability contract. All run on the shared nextest Postgres with SPACE_CI_REQUIRE_TEST_DEPS=1.

#	Test	New / extended / existing
L1	cluster-controller transient_lifecycle_sweeper::tests::postgres_backed::r17_candidate_select_resolves_routable_execution_fact: an unroutable 'k8s' restoring row lists routable_execution = false; a 'space' restoring row with no execution lists true	existing (both sides already covered)
C1	…::r18_converge_cas_requires_routable_execution_at_flip_time	extended: its raced row (routable execution retired, fresh creating generation with no pod IP) now has an identical 'space' twin. The Kubernetes sweeper's CAS claims nothing for the 'k8s' row; the Direct sweeper's CAS converges the twin to started.
P1	internal-api tests/published_state.rs publishing_owned_started_sandbox_leaves_unroutable_restoring_lifecycle	extended: an identical 'space' restoring row with the same creating execution, published started in the same census, converges; the 'k8s' row stays restoring
W1	api controllers::sandbox::wake_completion::tests::redrive_cas_refuses_unroutable_execution_at_flip_time	extended: a table over 'k8s' (redrive Abandoned, row stays restoring) and an identical 'space' row (Completed, started). The insert_k8s_sandbox helper now delegates to a managed_by-parameterised one.
K1	api controllers::substrate::kubernetes::tests::substrate_scope::builder_lookup_reads_only_kubernetes_nodes	new: two builder nodes, each with a node_k8s row and a ready builder node-api pod in the cache. The 'space' one is refused as "not registered"; the 'k8s' one gets past the lookup and dials a local listener. Bounded at 5 s, with the dial in an AbortOnDropHandle (the pattern of #506's D12).
D13, D14	api controllers::substrate::direct::tests::substrate_scope::create_target_claims_only_direct_rows	new: two creating, unplaced sandboxes that differ only in managed_by. The claim sets the 'space' row's target and refuses the 'k8s' row with Conflict, leaving it NULL. With both rows given the same target, the release clears only the 'space' row.
R1	api controllers::substrate::direct::tests::substrate_scope::route_identity_substrate_kind_reads_only_the_exact_labels	new unit table: "space" → Direct, "k8s" → Kubernetes; "direct", "kubernetes", "SPACE", "K8S", " k8s", "space ", "" → none

Test plumbing: the sweeper tests that run list_aged_transient_query/converge_restoring_query directly bind the extra $5 (6 call sites), and sandbox_routability/tests.rs passes the new argument.

Predicate removal

Each mutation changes one production site; then I ran the owning package's whole suite (cargo nextest run --locked -p <package> --retries 0 --no-fail-fast, SPACE_CI_REQUIRE_TEST_DEPS=1, --all-features for api) and reverted. P1a/P1b first ran without --all-features for internal-api and were rerun with it, with the same results; that crate's publication tests compile only with a feature. For a fragment call site there are two mutations, each made through the placeholder argument:

• (a) "$n OR TRUE": the predicate accepts every row.
• (b) "$n AND FALSE": the substrate exemption is dropped and the fragment reduces to the routability EXISTS.

"Before" is this PR's production code with main's tests plus the bind plumbing. "After" is this head.

#	Mutation	Tests that fail (after)	Before → after
L1a	list $5 OR TRUE	r17_candidate_select_resolves_routable_execution_fact	fails → fails
L1b	list $5 AND FALSE	same	fails → fails
C1a	converge $5 OR TRUE	r18_converge_cas_requires_routable_execution_at_flip_time	fails → fails
C1b	converge $5 AND FALSE	r18… (new twin), plus existing r10_converge_cas_respects_restore_lease and lifecycle_writer_tests::converge_cas_moves_only_restoring_rows	fails → fails (now also r18)
P1a	published $4 OR TRUE	publishing_owned_started_sandbox_leaves_unroutable_restoring_lifecycle	fails → fails
P1b	published $4 AND FALSE	publishing_owned_started… (new twin), plus existing published_state_tests::node_publication_converges_only_from_its_source_lifecycles	fails → fails (now also the twin)
W1a	wake CAS $2 OR TRUE	redrive_cas_refuses_unroutable_execution_at_flip_time	fails → fails
W1b	wake CAS $2 AND FALSE	redrive_cas_refuses… (new twin), plus existing lifecycle::characterization_tests::redrive_completes_only_a_restoring_row	fails → fails (now also the twin)
K1	builder lookup (n.managed_by = $2 OR TRUE)	builder_lookup_reads_only_kubernetes_nodes	passed → fails
R1a	Space → Kubernetes	route_identity_substrate_kind_reads_only_the_exact_labels, plus existing direct::tests::wake_noop_without_receipt_preserves_current_direct_identity and ingress::tests::stored_ingress_access_resumes_paused_execution_before_reporting_ready	fails → fails
R1b	unknown text → Direct	route_identity_substrate_kind_reads_only_the_exact_labels	passed → fails
D13	claim (managed_by = $3 OR TRUE)	create_target_claims_only_direct_rows	passed → fails
D14	release (managed_by = $3 OR TRUE)	same	passed → fails

On main's tests, 9 of the 13 mutations failed a test; on this PR's tests all 13 do. (For #458's dropped lookup, the mutation was also passed → fails.) Each extended test now fails both mutations of its statement on its own.

Plan cache: literal vs bound

Setup. A scratch database migrated from the test template on the repository's test image (PostgreSQL 16.15), with synthetic, deliberately skewed data:

• 400 nodes (10% 'space'; Kubernetes nodes with node_k8s rows; a builder every 50th);
• 200,000 sandboxes (5% 'space'; 30% deleted, 40% started, 15% suspended, 5% restoring, 5% stopped, 3% failed, 2% pausing), across 60 orgs, one of which holds 40%;
• placed Kubernetes sandboxes spread over about 330 nodes of about 333 residents each, plus three hot nodes of 3,333–3,556;
• an active execution per placed Kubernetes sandbox: running with a pod IP, except two thirds of the restoring ones, which are creating with no IP; and a retired generation for a third;
• then ANALYZE.

Each sequence ran on one connection with plan_cache_mode = auto: PREPARE once with the parameter types sqlx sends, then 30 × EXPLAIN (ANALYZE, TIMING OFF) EXECUTE (writes each in BEGIN … ROLLBACK, so the data stays fixed), then pg_prepared_statements. "Literal" is the statement text on main, "bound" this PR's.

Which statements. The fragment's Kubernetes-path callers:

• the published-state convergence, run in every node publication;
• the sweeper's aged-transient list, every tick;
• the sweeper's converge CAS, per candidate;
• the wake-completion CAS, per cancelled wake.

The builder lookup is also measured.

Sequence (30 executions, one connection)	Literal: generic / custom	Literal: planning sum; execution median, ms	Bound: generic / custom	Bound: planning sum; execution median, ms
P1: 3 hot + 27 ordinary Kubernetes nodes, shuffled, full census each	25 / 5	16.0; 1.46	25 / 5	15.9; 1.27
P1: the 3 hot nodes first, then 27 ordinary	25 / 5	17.0; 0.95	25 / 5	16.8; 0.88
P1: 30 ordinary nodes, censuses of 1–10 rows	25 / 5	3.4; 0.083	25 / 5	3.4; 0.080
L1: 30 ticks, age threshold 60–3,500 s shuffled	0 / 30	24.1; 48.5	0 / 30	25.3; 48.5
C1: 30 aged Kubernetes restoring rows, routable and not	25 / 5	3.0; 0.092	25 / 5	2.7; 0.070
W1: 30 Kubernetes restoring rows	25 / 5	2.5; 0.060	25 / 5	2.5; 0.062
K1: 30 lookups over 8 Kubernetes builders	25 / 5	1.1; 0.020	25 / 5	1.2; 0.023

D13/D14 (Direct-only) are primary-key updates. Their generic plans are identical apart from the filter text: (managed_by = $3) for (managed_by = 'space'::text) under the same id = $1 scan. On a Direct deployment's database sandboxes.managed_by is effectively single-valued, so 1/ndistinct equals the literal's frequency anyway (#506's argument).

• Plan choice is the same in every sequence, and planning and execution times match within noise.
  • The list statement builds a custom plan on every execution with the literal text too (its age threshold and random() order), so it already replanned on main.
  • Everything else latches the generic plan after five executions either way.
• The generic plans (EXPLAIN (GENERIC_PLAN)) are identical, costs and row estimates included, for all five Kubernetes-path statements. The only difference is the filter text, (managed_by <> $n) instead of (managed_by <> 'k8s'::text), in P1, C1 and W1, and (managed_by = $2) instead of = 'k8s' in K1.
  • In L1 the fragment sits in the select list, so the plan text doesn't differ at all.
  • In P1, C1 and W1 the predicate is a filter under a primary-key or id-driven index scan (Index Scan using sandboxes_id_org_id_key … Index Cond: (id = input.sandbox_id) for P1), ORed with the routability EXISTS, so its estimate can't change the access path.
  • K1 is a primary-key lookup on nodes.
• Indexes on managed_by (idx_nodes_managed_by, idx_nodes_role (managed_by, role), idx_sandboxes_managed_by) are full, not partial, and a <> filter can't use them. So a generic plan has no index it could lose.

No converted statement is worse bound, so no literal is kept as an exception. These are synthetic numbers on one PostgreSQL 16.15 instance; a production database's statistics decide the actual margins.

Characterization tests pass unchanged

The characterization suites pass in every run below:

• api adapter_characterization_tests (57) and lifecycle::characterization_tests (16, plus the 11 Kubernetes ones api: run the shared lifecycle tests on Kubernetes too #603 added), from api: characterize Kubernetes substrate suspend, pooled preparation and restore hydration (tests only) #273, tests: pin publication convergence and other lifecycle behavior from #272 #290, api: pin the pooled-setup error arms #273 called unreachable #326 and api: let tests inject the Kubernetes readiness and suspend budgets #330;
• the cluster controller's 21 lifecycle_writer_tests, from cluster-controller: characterize the lifecycle writers (tests only) #323;
• internal-api's 9 published_state_tests::lifecycle_writers (internal-api: characterize node publication's lifecycle writers (tests only) #324) and published_state_lifecycle_race_tests (internal-api: recheck the failed projection's guards under the row lock #337).

None of their files, nor lifecycle_writer_test_support.rs or scripted_test_doubles.rs, changes.

Docs agents read

• docs/rust/space-conventions.md, "Substrate values are bound parameters":
  • The paragraph listing remaining literals now says every crate's production SQL binds the value, and production Rust doesn't compare the label string or parse the running substrate's managed_by() to decide behavior (api, cluster-controller: make the Substrate variants private to the substrate module #607's rule; the substrate AGENTS.md names the one SQL-scope exception). It says pattern_ratchet.substrate_literals catches the common forms, not all, and points at its comment for the gaps.
  • A bullet after it carries Ask SubstrateKind row-level questions instead of matching the substrate #602's convention: a decision about a row's substrate calls a SubstrateKind method named for the question; repeating a statement's fixed scope compares against the named SubstrateKind const the statement binds; matching a variant is only for mapping it to a value; a fact about the running substrate reads capabilities(); never compare the label string. It is Ask SubstrateKind row-level questions instead of matching the substrate #602's bullet word for word, except that it says "named SubstrateKind const" and leaves out Ask SubstrateKind row-level questions instead of matching the substrate #602's examples (proves_execution_absence, restores_through_pods, REGISTRY_SUBSTRATE), which don't exist until Ask SubstrateKind row-level questions instead of matching the substrate #602 merges.
  • Reconciling with Ask SubstrateKind row-level questions instead of matching the substrate #602, which edits 10 lines of this file and 6 of the substrate AGENTS.md. Its space-conventions.md hunk is anchored on the "direct.rs has about a dozen" text this PR deletes, so whichever lands second gets a conflict there. Keep this PR's paragraph and Ask SubstrateKind row-level questions instead of matching the substrate #602's bullet with "named SubstrateKind const". With that one-word change, Ask SubstrateKind row-level questions instead of matching the substrate #602's bullet is a superset of this PR's bullet, which is dropped. Ask SubstrateKind row-level questions instead of matching the substrate #602's AGENTS.md bullet goes above the "Scoping a query" bullet this PR edits and doesn't overlap it.
  • "direct.rs has about a dozen" and the fragment-as-debt text are gone.
  • The remaining named debt is the sandbox store's text SandboxRecord.managed_by/AutoLifecycleExecution.managed_by fields (still String on main).
  • The row_state_guards sentence now also names the routability fragment as an existing fragment that takes its placeholder, not a pattern for new code.
• crates/libs/api/src/controllers/substrate/AGENTS.md: "Existing literals, such as the shared routable_execution_or_non_k8s fragment, are debt" → the archcheck rule catches the common literal forms in production code, and its comment lists what it misses. The rest of api, cluster-controller: make the Substrate variants private to the substrate module #607's new text in that bullet is kept.
• crates/libs/cluster/controller/server/AGENTS.md: unchanged. On main it already says to scope with managed_by = $n, never a literal, bound from substrate.managed_by() or a fixed SubstrateKind (the Kubernetes arm of run_reconcilers). No text anywhere still tells a Kubernetes-only loop to write managed_by = 'k8s'.
• The space-rust-development skill: its "remaining raw-literal sites … are existing debt" sentence is replaced. In Rust, a decision about a row's substrate calls the SubstrateKind method named for the question, a fact about the running substrate reads capabilities(), and code never compares the label string; it points at the conventions subsection. archcheck's rule catches the common literal forms in production code. The plugin version goes from 0.2.33 to 0.2.34 in plugin.yaml and both plugin.json files (.agents/plugins/AGENTS.md). claude plugin validate passes; its one warning, a missing author, is also on main.

Decisions made unattended

Decision	Rationale
Keep routable_execution_or_non_k8s a fragment that takes the placeholder	The task names this shape. Inlining would give the routability contract four copies. The conventions doc names it next to row_state_guards as an existing fragment, not a pattern.
Bind the fixed SubstrateKind::Kubernetes everywhere	The fragment asks "is this row Kubernetes?", whichever substrate runs: the Direct sweeper binds $1 = 'space' and $5 = 'k8s', as the literal did.
Convert the literals #217 and #458 added while this PR was open	The rule is required, so the rebased head would otherwise fail its own check. Each conversion got a twin test and a mutation, as the others did. #458's was dropped again when #606 reverted it.
Rerun Kind in otel mode	CI's Kind shards pin SPACE_KIND_TELEMETRY_MODE: otel (kind-ci.yml). My first runs used the harness default (datadog), which skips two otel-only scenarios, so the later runs match CI.
Extract route_identity_substrate_kind in direct.rs	The DB CHECK constraint makes the 'k8s' and unknown arms unreachable through Postgres, so only a unit table can pin that the parse is identical for every input. The extraction is 16 lines beside parse_direct_route_identity. The call site keeps the same error and message.
Extend existing tests rather than add new ones for L1/C1/P1/W1	Each existing test already held one side; an otherwise identical twin of the other substrate in the same test is the smallest change that makes both mutations fail it. The test-economy rule prefers extending a table over adding a function.
SQL patterns name managed_by only; Rust patterns name managed_by/substrate_kind values and match arms	sbx-mgr's SQLite substrate_kind is a node-local notion the plan keeps separate. Matching any = 'k8s' would flag it, and Kubernetes-y strings in unrelated SQL.
no_crossing, listed_no_growth, stale_entries required; no_new_entries, no_raised_caps advisory; no baseline table	Zero matches on the converted tree and no false positive on main. The two advisory modes give a false positive a reviewed path (a baseline entry) instead of a silent allow widening, as in every sibling instance (review round 1).
Arm patterns need a substrate name on the line	Without it, any "k8s"/"space" arm counted, including non-substrate vocabularies. The cost is an arm whose substrate name is only on the match line; M1/M2 catch that match header when it matches the label string, which is how such an arm arises.
Allow-list the known test-support globs rather than individual files	They are the repository's #[cfg(test)] support naming conventions (*_tests/, test_support, *_test_support.rs, test_fixtures.rs, *_test_fixture.rs, scripted_test_doubles.rs, the db-test-fixtures crate). Narrower globs would turn the next fixture into a required failure. Generic names (fake, transport, compatibility) are not allowed.
AGENTS.md for the cluster controller unchanged	Its text already says to bind (updated by #373).
Kind: add Docker's loopback masquerade rule for the Kind bridge	See Validation. The host's Docker didn't install it, so kubectl to 127.0.0.1 failed. Scoped to the task's own Kind network and removed at each teardown.

Locator note

The converted statements' text changes (a placeholder replaces the literal), so their sqlx statement-cache keys and query signatures change. Statements below an added .bind line move to new /*module_path:line*/ locators. Module paths don't change.

Overlaps

• api, cluster-controller: make the Substrate variants private to the substrate module #607 (phase 3) merged during this PR. The rebase conflicted only in ARCHITECTURE.toml (both added instances; both kept) and the substrate AGENTS.md bullet (api, cluster-controller: make the Substrate variants private to the substrate module #607's new wording kept, with this PR's sentence about literals). git range-diff shows no other change to this commit.
• Ask SubstrateKind row-level questions instead of matching the substrate #602 (row-level SubstrateKind methods) is open. It edits docs/rust/space-conventions.md (10 lines) and the substrate AGENTS.md (6 lines). See "Reconciling with Ask SubstrateKind row-level questions instead of matching the substrate #602" above for the trivial resolution.
• This PR touches no archcheck code; the rule is configuration only.

Review round 1 → changes

Finding	Change
MINOR: the rule has no reviewed-exception path (no_new_entries/no_raised_caps required, unlike the sibling instances)	Both are advisory. The instance comment says a deliberate exception is a reviewed baseline entry. no_crossing, listed_no_growth and stale_entries stay required.
MINOR: false positives (any "k8s"/"space" arm; comments)	Arm patterns A1/A2 now need a substrate name on the line. The remaining false-positive classes are listed in the instance comment: trailing ////* */, #[doc], SQL -- comments, and a non-substrate vocabulary sharing a line with a substrate name. No production line matches on this head; on main only the converted ones do.
MINOR: false negatives	Now caught: matches!(…managed_by…, "k8s"), .as_deref() == Some("k8s"), Some("k8s") =>, "k8s" if … =>, ("k8s", true) =>, eq/eq_ignore_ascii_case("k8s"), prefixed names, IN (lower($1), 'k8s'), = ANY(ARRAY['k8s']), same-line CASE managed_by WHEN 'k8s', managed_by::text = 'k8s', lower(managed_by) = 'k8s', and a match on the label string. Still missed, and listed in the comment: VALUES lists, split lines, format!, non-.rs SQL, an arm with its substrate name only on the match line, and a guard containing =/::. Probe table above.
MINOR: docs overclaim and diverge from #602/#607	space-conventions.md, the skill and the substrate AGENTS.md say the rule catches the common forms. They now use #602's and #607's conventions (SubstrateKind method for a row decision, the named const for a fixed scope, variant matching only for mapping, capabilities() for the running substrate, no parse of the running managed_by(), never the label string). "Parse the value and match the variant" is gone. The text is shaped so #602 reconciles trivially.
NIT: the fragment's placeholder is free-form	The doc comment says it must be a placeholder used nowhere else in the statement, and that reusing one (the sweeper's $1) would change which rows the fragment exempts.
NIT: PR body on #602 docs; instance comment vs body	Corrected (#602 edits 10 + 6 doc lines). The body's limits list is the instance comment's.

This round changes ARCHITECTURE.toml, docs and one doc comment; no Rust code. So nextest and Kind weren't rerun; their rows below are from 38caefa, whose code is identical.

Review round 2 → changes

Finding	Change
MINOR: comparing the label through a named const is an unlisted bypass	C1/C2 catch == NODE_MANAGED_BY_K8S/_SPACE (any path prefix) and == SubstrateKind::…/NodeManagedBy::….as_str()/.to_string(), either side. No hits anywhere on this head, tests included. The parser arm NODE_MANAGED_BY_K8S => … and .bind(SubstrateKind::Kubernetes.as_str()) don't match. The consts stay public. The conventions bullet now says "the named SubstrateKind const the statement binds".
MINOR: the match-header pattern fires on any substrate-named identifier	M1's identifier is narrowed to *managed_by/substrate_kind. match substrate_outcome.as_ref() {, match self.substrate.as_ref() { and match substrate_capabilities.as_ref() { no longer match. main's match substrate_kind.as_str() { still does. .as_ref()/.as_deref() on a typed Option<SubstrateKind>/Option<NodeManagedBy> is listed as a false positive.
MINOR: quoted identifier, 'k8s' IN (n.managed_by) and raw strings missed	All three are now caught: S1/S2/S3/S5 accept "managed_by", S4 handles 'k8s' [NOT] IN (…managed_by…), and R1–R3 accept r"…"/r#"…"#. The comment and body say "either side" applies to the binary comparisons only.
NIT: complete the comment's lists	Added misses: if let/let … else, [...].contains(…), a column read by name, a neutral binding, a case-folded literal, the &*/[..]/.unwrap_or(…) headers and their arms, and any ;{}=: between an arm's literal and =>. Added false positive: SQL-shaped text in an ordinary Rust string. Every one is in the probe.

Validation

Head 9a18529 on main 9af70ec. Rounds 1 and 2 changed config, docs and one doc comment only. Before them, head 38caefa. The commit was rebased six times as main moved: c1187fb → 125c08f → 04c03e6 → 17cd654 → 25cfbf8 → 202ef6b → 9af70ec. The rebase onto 17cd654 added the #217/#458 conversions; the one onto 25cfbf8 dropped #458's after its revert. Each row names the heads it ran on. Rounds 1 and 2 changed no Rust code apart from one doc comment, so nextest and Kind weren't rerun after 38caefa. Kind ran last on 77f5a3b, this commit on 25cfbf8. git range-diff shows this head differs from it only in the two conflict resolutions with #607 (ARCHITECTURE.toml, the substrate AGENTS.md), so the production diff Kind validated is identical. The last rebase, onto #603 (tests only), left the commit unchanged (range-diff: =). #607's own production changes (Substrate variants private), #610 (CI) and #603 (tests) are not covered by these Kind runs.

Check	Result
cargo fmt --all -- --check	pass (9a18529, 711ff14 and 38caefa)
cargo clippy --workspace --all-targets --all-features --locked -- -D warnings	pass (9a18529, 711ff14 and 38caefa)
cargo clippy -p pplx-space-api --no-default-features -- -D warnings (Direct-only build; direct.rs changed)	pass. With --all-targets it reports one unused import in controllers/sandbox/mod.rs (pooled_prepare_error_is_reclaimable) that is only unused in test builds without kubernetes. That file is untouched: PRE-EXISTING.
pre-commit (Gitleaks) at commit, and pre-commit run gitleaks --from-ref origin/main --to-ref HEAD	pass (9a18529, 711ff14 and 38caefa)
space-public-review on the staged change and this text	no findings in the reviewed scope (loopback and test-net addresses, synthetic names; the node-api port annotation key is the existing public constant in caches.rs)
cargo arch --base-baseline <main's baseline> --base-architecture <main's ARCHITECTURE.toml>, and --format rdjson as CI runs it	exit 0 for both on 9a18529. 90 advisory and 0 required findings, the same set as main. On main with this ARCHITECTURE.toml: 3 required substrate_literals findings (the 7 lines above). Growth in three files already over advisory listed_no_growth caps: direct.rs 3981 → 3992 (route_identity_substrate_kind and two .bind lines), substrate_adapter.rs 2506 → 2508, published_state.rs 3806 → 3807.
cargo nextest run --locked -p pplx-space-api --all-features -p pplx-space-cluster-controller-server -p pplx-space-internal-api-server -p pplx-space-common-svc --run-ignored all --retries 0, SPACE_CI_REQUIRE_TEST_DEPS=1, three runs	38caefa: 3478/3478 passed, 0 skipped, in each of runs 1, 2 and 3. Every run includes the characterization tests above, #603's 11 new Kubernetes lifecycle characterization tests and the six new or extended tests. Earlier heads, three runs each: d8d95ba 3453/3453, 58523c8 3453/3453, 5efa6db 3480/3480, 77f5a3b 3456/3456, eb0d1f9 3457/3457.
claude plugin validate .agents/plugins/pplx-space	pass (one warning, missing author, as on main)
archcheck's own tests (cargo nextest run --locked -p pplx-space-archcheck, includes tests/repository.rs against this ARCHITECTURE.toml)	55/55 passed on 9a18529
Synthetic probe	52/52 positives on their own pattern; 0 of 35 look-alikes and 0 of 11 listed misses matched; both listed false positives matched; archcheck counted 54 (table above)
Predicate removal; plan cache; rule matches	see Evidence

Kind

Step	Result
Disk before each bring-up	Docker data device 17–35 GB free. It dropped to 4–10 GB during typed suites, with no disk-related failure. Cleanup (Disk section): (1) with no cluster running and 14 GB free, docker builder prune -af freed 20.4 GB on the device and docker image prune -f 0 B, leaving 35.1 GB. (2) This worktree's target/, removed three times after the cargo validation for the pushed head had finished, freed 86.6, 78.0 and 35.0 GB on the home volume (see the builder-storage item below). No later step was needed.
ENVIRONMENTAL: Docker loopback port publishing	On this host, Docker installs the 127.0.0.1 port DNAT for user-defined bridges but not the matching -o <bridge> -m addrtype --src-type LOCAL -j MASQUERADE it has for docker0. So kubectl to the Kind apiserver on 127.0.0.1:<port> failed ("no route to host", then connection refused), and the first bring-up failed in check-object-store-transition.sh. Recreating the kind network didn't add the rule. For each run I added that one rule for the task's own Kind bridge with iptables -t nat, and removed it at teardown. Nothing else on the host changed.
Fresh cluster, build provenance (final run, 77f5a3b)	No cluster existed. smoke-kind-e2e.sh (BOOTSTRAP_TIMEOUT=1800, DATA_PLANE_URL, SPACE_KIND_TELEMETRY_MODE=otel as CI sets) created space-local (control plane created 04:25 UTC) and started Tilt from this checkout. The api, internal-api and cluster-controller services log version: 77f5a3bc77f6. Their deployed binaries match the artifact image's by sha256 (cc8be0aa…, 212223a4…, cd12fa30…) before and after the suites. None contains managed_by <> 'k8s', managed_by = 'k8s' or managed_by = 'space'. The api binary contains the builder lookup's n.managed_by = $2 and the create-target managed_by = $3 AND lifecycle = 'creating'.
smoke-kind-e2e.sh (start)	pass (runtime-manager checks and 7/7)
smoke-kind-typed.sh all (unsharded, CI's env: otel, admin token file, cold-receipts dir, host dm targets loaded)	ECW 77/78, native 24/24, 0 warned, inconclusive or quarantined. The one failure is below.
smoke-kind-e2e.sh (end)	pass (runtime-manager checks and 7/7)
Cleanup	teardown.sh on each task-owned cluster; the iptables rule removed; scratch Postgres containers and scratch worktrees removed

Typed-suite failure on the final run, rerun once: PRE-EXISTING. local.btrfs_send_restore_pulls_parent_template: "template-manager counted no successful on-demand pull of the parent on the worker; the build may have come from prefetch". The rerun failed the same way. On a fresh otel stack built from pristine main 17cd654 (no commit of this PR), the same scenario fails the same way. It passed in both otel runs at 58523c8 (base 04c03e6), so it arrived with one of main's commits between 04c03e6 and 17cd654; I did not bisect which. This PR touches neither template-manager nor the pull path.

Earlier Kind runs

• d8d95ba (base 125c08f, harness default telemetry datadog), fresh cluster, provenance as above (version: d8d95babec4e): e2e 7/7, typed ECW 78/78 and native 23/23, e2e 7/7. In datadog mode the suite skips local.btrfs_send_restore_pulls_parent_template and local.telemetry_reaches_local_backends, so the later runs used CI's otel.
• 58523c8 (base 04c03e6): first a datadog run (e2e 7/7, ECW 76/76, native 24/24, e2e 7/7). Then two fresh otel runs:
  • The first had ECW 77/78 with one ENVIRONMENTAL INFRA_ERROR. local.ecw_template_build_from_oci_boots hit "storage admission requires 55834574848 available bytes but found 51032117248": the builder's storage is on the home volume, which this worktree's target/ had filled. I tore down, removed target/ (Disk step 2) and reran on a fresh cluster.
  • The rerun: e2e 7/7, ECW 78/78, native 24/24, e2e 7/7.
• 5efa6db (base 17cd654, while Enforce tenant publication grants after preparation rollout #458 was on main), fresh otel cluster: e2e 7/7, ECW 75/78, native 24/24, e2e 7/7. The three failures were all PRE-EXISTING: each reproduced on the pristine main 17cd654 stack.
  • local.ecw_template_build_from_oci_boots and local.private_template_build_isolation were refused at registration with "publisher node UID is awaiting inventory synchronization". That is Enforce tenant publication grants after preparation rollout #458's new check, and the Kind builders' node_k8s.k8s_node_uid was NULL on both stacks. That message means the lookup found the builder row, so its managed_by predicate matched. Run against the live Kind database with the builder's identity, main's literal statement and this PR's bound one returned the same row. Both pass again on 77f5a3b, after Revert "Enforce tenant publication grants after preparation rollout (#458)" #606's revert.
  • local.btrfs_send_restore_pulls_parent_template: the failure above.

🤖 Generated with Claude Code