[ac-pplx]

What merging this does

Tests only; no production code, schema, wire format, metric or configuration change. #402 sorted the tests that build a DirectSubstrate and listed 23 api tests (bucket 4) whose shared-lifecycle outcome or path differs on Kubernetes. This PR adds a Kubernetes-backed twin for 21 of them and keeps every Direct test.

• Each twin keeps its Direct twin's setup and assertions. Its doc says whether the Kubernetes outcome is the same and, where it differs, why. Where the Kubernetes outcome differs, the twin asserts it.

• Four twins are renamed because the Direct name's claim is false on Kubernetes. Their docs name the Direct test.

• Where the twins live. Each sits in a kubernetes submodule beside its Direct test file, compiled only with --features kubernetes. The exception is live_cpu_update_preserves_allocation_across_cold_restart, whose Direct copy is under substrate/direct/. Its twin went to updates/postgres_tests/kubernetes.rs, the home of the shared update path it pins.

• What they're built from. They use the kubernetes::tests substrate fixtures and the scripted apiserver double (scripted_kube_client).

• Shared fixtures are in test_support::kubernetes:

  • an in-memory pod-store apiserver;
  • a live sandbox: an active sandbox_k8s_executions row, plus the Running pod that the reflector and the apiserver both hold, named as the substrate names generation 1's pod;
  • the node-api simulator registered as a Kubernetes worker node.

  That module is the only place these tests construct a Substrate (Substrate::from_kubernetes, since api, cluster-controller: make the Substrate variants private to the substrate module #607 made the variants private).

• Route shape of a seeded execution. By default it has published its route: route_activation_id names the attempt the resident reports, so activations take PreserveExisting, as for executions the current api activated. Only the tests that hook the route publication seed the legacy, route-less shape (RouteIdentity::Legacy): the wake-completion twin, the native-start twin and the start-after-Wake arm of completed_actions_keep….

• Fixture limitations, stated in their docs. The reflectors are seeded once. Pods the apiserver double later creates or deletes reach the substrate only through its uncached reads, and every pod is at the simulator's address. The double creates pods already Running and, with no scheduler, never binds one created without spec.nodeName to a node. It deletes pods at once, with no Terminating phase. Like a real apiserver, it refuses with a 409 a delete whose preconditions.uid names another pod; it does not check preconditions.resourceVersion.

• Changes outside the new files:

  • one #[cfg(feature = "kubernetes")] mod kubernetes; line in each of seven test files and in test_support.rs;
  • substrate_with_store_client_proxy_port_and_pods in kubernetes/tests.rs, a multi-pod form of the existing single-pod fixture. The Kubernetes suspend and observed-only delete paths need the node's node-api pod and a sandbox pod in the reflectors at once.
  • In public_sandbox_operations/tests.rs, the create-operation setup is extracted into seed_unpinned_create_operation, so both substrates' create-retry tests share it. The Direct test is otherwise byte-identical.
  • doc updates to the characterization module doc, the Direct key-share test, the updates/postgres_tests.rs scope note, and api, cluster-controller: sort the tests that use the Direct substrate as a fixture (tests only) #402's placeholder comment above public_sandbox_operations::tests::direct_sandbox_controller.

• Test count: api 1833 → 1854. That is 21 new tests, with the one #[ignore] benchmark still skipped.

Key-share finding

completed_actions_do_not_wait_for_a_key_share_lock holds on Direct only. Its premise: any foreign-key insert into a child table takes FOR KEY SHARE on the sandbox row, and a completed action should not wait behind it. On Kubernetes, these actions block on a statement whose row lock conflicts with FOR KEY SHARE, while an open transaction holds that lock on the row:

action	blocks?	statement it blocks on (sql! module)	what in the api bounds the wait
start (stopped row)	yes	SELECT lifecycle FROM sandboxes WHERE id = $1 FOR UPDATE, run by lock_sandbox_execution_lifecycle_in (kubernetes::store::execution_rows)	the activation wait budget (ready_wait_bounds) around ensure_pod_exists in start_execution
wake (paused row)	yes	same	the same budget, in drive_prepared_wake
stop, kill (pod gone)	yes	SELECT id FROM sandboxes WHERE id = $1 FOR UPDATE, run by lock_sandbox_for_execution_mutation_in (kubernetes::store::execution_rows)	nothing: the wait is in pod resolution, before ensure_execution_ready_for_access takes its budget
suspend	yes	SELECT id FROM sandboxes WHERE id = $1 FOR UPDATE (kubernetes::store::suspend; the only statement of that shape there, in persist_or_accept_suspend_commit). This is the counterpart of Direct suspend's artifact lock.	nothing: the suspend_durability deadline covers the node-api resolve, connect and suspend poll, not finish_suspend_handoff; the suspend also runs in a tracked task that outlives its request
resume, pause, stop, kill (pod cached)	no	n/a	n/a

• Which caller blocks. The sql! locator records where a statement is defined, not which caller ran it. lock_sandbox_execution_lifecycle_in has three direct callers: persist_materialized_pod_metadata, claim_started_recovery and lock_sandbox_for_execution_lifecycle_in. The last is reached from both the execution claim and claim_active_execution_pod_materialization. That start's and wake's first wait is the execution claim ensure_pod_exists takes is an inference from the code. lock_sandbox_for_execution_mutation_in has many callers; that the missing-pod stop and kill wait in the pod-tracking cleanup of the missing pod (resolve_pod_instance → forget_pod_tracking_if_generation), before their recovering client recreates the pod, is likewise read from the code.
• What the test shows. The blocking transaction ends promptly, and each waiter then completes. Kubernetes suspend then ends unplaced, where Direct suspend leaves the row on_node. The missing-pod stop and kill recreate the pod (one pod create each) and end stopped/unplaced.
• A longer blocker. It would not be waited out indefinitely. The api sets neither lock_timeout nor statement_timeout on its own sessions, but it connects as space_user, whose role-level statement_timeout bootstrap-roles sets (--app-statement-timeout, 10s by default). That ends every wait in the table with an error. The role's lock_timeout is opt-in (SPACE_DB_APP_LOCK_TIMEOUT). Within the api, only start and wake have a deadline of their own around the wait (last column).

The twin activations_and_suspend_wait_for_a_key_share_lock pins this deterministically: no #[ignore] was needed. Each wait is tied to the holder's backend pid ($holder = ANY(pg_blocking_pids(pid))) and to the waiting statement's sql! locator. Every wait and join is bounded. Not fixed here.

Test → Kubernetes outcome

Direct test → Kubernetes twin	Kubernetes outcome	why
characterization_tests::wake_refuses_teardown_and_terminal_lifecycles_before_admission	same refusals; the passing lifecycles are pinned	The shared entry guard refuses the same lifecycles with the same 409. The Direct test only asserts that the passing lifecycles get something other than 409; this twin pins each one. Entry admission refuses created with a 429. restoring and suspended defer their charge to the restore-lease claim, which refuses with the same 429 and releases the claim; the row stays as it was. The substrate refuses a suspending lifecycle with a 503 (suspend finalization in flight). The others reach the substrate's pod create, which this double's apiserver refuses (a 500, as in the next row).
…::unreachable_runtime_fails_actions_without_writing_lifecycle	different	No action writes a lifecycle, as on Direct. Wake, stop and kill take the recovering path, which tries to recreate the pod; the apiserver's 503 on the create is followed by an uncached read that finds no pod, so they fail as a 500 (failed to create sandbox pod). If that read also failed, the result would be a 503 with a retry hint, so the 500 belongs to this double, not to every outage. The rest are 503s without a pod create: • start of a stopped row claims a new execution, finds no restore source and abandons the claim; • suspend finds no execution; • pause and checkpoint use the non-recovering client; • resume's recovering client does not recover, because paused is dormant.
…::wake_completion_outcome_maps_to_the_wake_response	different at the Direct injection point	Kubernetes re-reads the row in ensure_pod_exists before the node Wake. A row moved after the handler's read is refused there: suspending → 503, teardown or terminal → 409 (cannot materialize an execution after lifecycle changed …). Direct answers 200 and a CAS-lost 500. A second pass moves the row after the node acknowledged the Wake, from a trigger on the route publication; there the completion-CAS mapping is the same as Direct's.
…::completed_actions_keep_a_terminal_lifecycle_that_landed_mid_flight	same when the lifecycle lands after the node call; start and suspend refuse one that landed before it	The row keeps the terminal lifecycle in every case. Landed before the action, as in the Direct test: resume, pause, stop and kill report success and reach the controller's guarded write, as on Direct. Start and suspend re-read the row first, so they refuse it before the guarded write: start with a 409, suspend's claim CAS from started/paused to suspending (claim_suspend_execution) with a 503. So start and suspend also land it after the node call. Start's lands once the node acknowledged its Wake, from the route-publication trigger (a RouteIdentity::Legacy execution). Suspend's lands with the substrate's own suspended finalization, from a trigger on that write. Both then report success and the controller's guard keeps the lifecycle. After start's Wake the row ends on_node: the substrate records the pod's node placement itself (persist_pod_node_placement). Between suspend's claim and its finalization, the substrate's own writes accept only a suspending row; no arm lands a lifecycle there.
…::native_start_overwrites_a_stopped_written_mid_flight_without_a_retry	same	The stopped is written by a trigger on the legacy route publication (route_activation_id), the Kubernetes counterpart of the Direct route-receipt trigger. The shared, unguarded start write then overwrites it, and the operation succeeds without a retry.
…::completed_actions_do_not_wait_for_a_key_share_lock → activations_and_suspend_wait_for_a_key_share_lock	different	Start, wake, suspend, and stop and kill of a sandbox whose pod is gone block (see the finding above). Suspend ends suspended/unplaced, not Direct's suspended/on_node.
…::stop_and_kill_on_a_draining_node_still_reach_the_worker	same, plus a Kubernetes-only arm	Stop and kill reach the worker through the live pod. The added arm: for a sandbox whose pod the apiserver no longer has, the non-recovering client finds no runtime, so the action is a 503 and does not recover a pod onto the draining node. The lifecycle is kept, but resolving the missing pod forgets its tracking, which unplaces the row.
…::untracked_runtime_actions_without_placement_are_unavailable → untracked_runtime_actions_without_a_row_are_not_found	different	The Kubernetes substrate looks the id up itself (require_sandbox), so an id with no row is a 404. Suspend reads the active execution first: a 503. No request reaches the apiserver.
…::untracked_actions_report_status_and_leave_the_row → untracked_actions_report_status_and_leave_the_row_unless_suspend_claims_it	different for suspend	The other actions report their status and leave the row, as on Direct. Kubernetes suspend takes the same path tracked or not: it claims the row with its CAS and its handoff writes it. So on a created row it is a 503 before reaching the worker, while on a started or paused row it suspends the resident and writes the row suspended/unplaced.
…::observed_only_delete_removes_the_resident_on_the_observing_node	same	no warm-pool slot is reserved, so the delete resolves the node-api pod from the node cache
…::redrive_completes_only_a_restoring_row	same with a routable execution, plus a Kubernetes-only arm	Direct's hydration probe always answers routable. On Kubernetes, a restoring row whose execution has no pod IP, and whose pod the apiserver lacks, stays restoring.
lifecycle::tests::exhausting_the_finalize_retry_budget_stays_retryable	same	the Kubernetes teardown finds no execution, and the finalize loop is shared
lifecycle::tests::a_contended_delete_precheck_stays_retryable	same	The pre-check runs before the substrate. The Direct test accepts any retry outcome but contention; this twin asserts that the retry deletes the row, through the Kubernetes teardown (no execution to retire) and the shared finalize.
lifecycle::tests::resume_uses_resolved_runtime_without_async_observation	same	the runtime client comes from the active execution's pod
ingress::tests::stored_ingress_access_resumes_paused_execution_before_reporting_ready	same	the route identity is sandbox_k8s_executions.route_activation_id, and the resume goes through the pod
create_postgres_tests::postgres_failed_pooled_create_finalizes_after_claim_cleanup → postgres_failed_pooled_create_defers_cleanup_to_the_reconciler	different	cleanup_failed_pooled_prepare marks the claimed slot cleanup_pending and returns DeferredToReconciler, so the row is kept. On Direct the row is deleted.
public_sandbox_operations::tests::create_executor_retries_unpinned_transient_substrate_failure	same	The operation is queued for retry either way. Direct finds no worker (a 503). On Kubernetes the apiserver's 503 on the pod create becomes a failed create, a controller 500. Native create execution retries any controller 5xx other than a known template-snapshot incompatibility (controller_error_result).
public_sandbox_operations::tests::admitted_create_re_execution_after_success_does_not_create_again	same with a claimed slot; different with cleanup_pending	Kubernetes admitted_pool_cleanup_pending reads the reserved slot, so a cleanup_pending slot gives workflow progress instead of Succeeded; on Direct that answer is always false. The Succeeded check is what catches a re-execution that re-enters create. The 0-executions check stays zero even then: the fixture tags no template release latest, so such a create fails resolving the template before it reaches the substrate.
updates::transport_tests::untransmitted_update_is_replayed_once_on_a_fresh_client	same	the fresh client is resolved through the active execution's pod (probe_runtime_client) instead of node_direct
updates::transport_tests::interrupted_update_is_observed_not_resent	same	the observer reaches the restarted worker through the pod
substrate::direct::tests::live_cpu_update_preserves_allocation_across_cold_restart → updates::postgres_tests::kubernetes::…	same	The cold restart retires the pod and creates generation 2's (…-g2), which carries space.pplx.net/vcpu: "8". The row's allocation stays 8. The update after the restart reads that -g2 pod from the apiserver, which shows it resolved the new pod. The simulator can't show which pod a call reached, because every pod shares its address.

Not covered (left for later PRs)

• template::tests::build_submission_grants_the_base_before_dispatch and system_build_withdraws_its_publication_grant_when_dispatch_fails. The Kubernetes builder client always dials the fixed in-cluster name space-builder-node-api:9800, with connect retries. A deterministic dispatch failure would need host DNS state or a production seam, and a tests-only PR can't add the seam.
• cluster-controller::transient_lifecycle_sweeper::lifecycle_writer_tests::converge_cas_moves_only_restoring_rows (5→4). The cluster controller has no KubernetesSubstrate fixture yet; api, cluster-controller: sort the tests that use the Direct substrate as a fixture (tests only) #402's comment there is unchanged.
• The roughly 16 test files that get Direct implicitly through AppState::from_pool / SandboxController::from_env (out of api, cluster-controller: sort the tests that use the Direct substrate as a fixture (tests only) #402's scope too).

Other characterization notes (pinned, not fixed)

• A failed pod create on a recovering wake, stop or kill. When the apiserver refuses the pod create with a 503 and its follow-up uncached read finds no pod, the result is a controller 500 (SubstrateError::Failed). If that read failed too, it would be restore_in_flight, a 503 with Retry-After. Native operation execution retries 5xx either way.
• On a draining node, a stop or kill that finds the pod gone unplaces the row (forget_pod_tracking_if_generation) while refusing with a 503.
• A stop or kill of a sandbox whose pod is gone waits behind FOR KEY SHARE in pod resolution, with no api deadline around that wait.
• Kubernetes suspend's commit lock wait has no api deadline either; only the role's statement_timeout ends it.
• An untracked suspend of a started or paused Kubernetes sandbox writes its row suspended/unplaced.

Decisions made unattended

1. Twins in sibling kubernetes submodules rather than parameterizing each test over both substrates. The setups differ structurally: worker registration through node_direct versus an execution row plus pod plus apiserver. Parameterizing would interleave the two and hide each one's outcome.
2. Rename only where the Direct name states something false on Kubernetes (four twins); every other twin keeps its Direct twin's name.
3. Shared fixtures in test_support::kubernetes, with one Substrate construction site (Substrate::from_kubernetes since api, cluster-controller: make the Substrate variants private to the substrate module #607).
4. A multi-pod fixture variant added to the existing private builder, instead of a second cache-seeding path.
5. A route-publication trigger as the mid-flight injection hook. The tests that use it seed the legacy, route-less execution, so the activation publishes its route right after the node acknowledged its Wake (publish_legacy_route_identity_after_ack), before the controller's lifecycle write. This mirrors the Direct tests' route-receipt trigger. Every other test seeds a published route. Suspend has no route publication, so its mid-flight hook is a trigger on the substrate's suspending → suspended write, in the same transaction. The finalization holds the row lock until it commits, so this is the write a concurrent writer would make right after it.
6. Key-share: a deterministic, pid-tied pin, not #[ignore].
7. Redrive mutations. The redrive twin's unroutable arm is guarded twice: by the probe and by the converge CAS's routable-execution SQL term. Its single mutation therefore breaks the routable arm. A double mutation of both guards was also checked, and it fails the unroutable arm.
8. Placement of the mod lines. They go beside each file's use block, or above the Direct fixture helper in updates/postgres_tests.rs, instead of at EOF, so that open PRs touching the same files still apply.
9. wake_refuses… uses the scripted unavailable apiserver rather than the inert client, so its pod-create refusal is deterministic.

Review follow-up

Each comment was re-checked against the code first. Where a premise did not hold, the change follows the code, and the row says what differed.

comment	change
Key-share "stop, kill: no" holds only with a cached pod	Added missing-pod arms for stop and kill, pid-tied and bounded like the others; they block, recreate the pod (one create each), then complete stopped/unplaced. The cached-pod "no" is scoped in the doc and the table. Premise partly different: the missing-pod stop and kill do not first wait on the materialization SELECT lifecycle … FOR UPDATE that start waits on. Their first wait is SELECT id FROM sandboxes WHERE id = $1 FOR UPDATE (lock_sandbox_for_execution_mutation_in), which, reading the code, the pod-tracking cleanup of the missing pod runs in resolve_pod_instance, before any recovery. A probe of pg_stat_activity during the wait showed that statement; the arm waits on it.
completed_actions_keep…: start and suspend never reach the guarded write	Start also lands the terminal lifecycle after the node acknowledged its Wake (route-publication trigger, RouteIdentity::Legacy seed). Suspend has a test-only seam after its claim: a trigger on its own suspending → suspended finalization. Both now report success and reach the controller's guard. Dropping the guard from start's maybe_update_sandbox_state fails the new twin and passes the old one. Exempting a suspended write from maybe_update_sandbox_lifecycle's guard does the same for the suspend arm. The up-front refusal arms are kept.
Name the bound that applies to each wait	Doc and table name it per action: start and wake, the activation wait budget around ensure_pod_exists; the missing-pod stop and kill, nothing in the api; suspend, nothing in the api (its suspend_durability deadline does not cover finish_suspend_handoff). Premise partly different: the api sets no session timeout, but it connects as space_user, whose role-level statement_timeout bootstrap-roles sets by default (--app-statement-timeout, 10s). So by default every one of these waits is ended by that role setting. The role's lock_timeout is opt-in, as the comment says.
a_contended_delete_precheck… retry assertion was the Direct weak form	The twin asserts that the retry succeeds and that the row is gone. Mutation: the Kubernetes teardown returns a deferring SuspendInFlight; the new twin fails and the old one passes. A teardown returning SubstrateError::Failed does not fail either twin, because the controller logs such a teardown failure and still deletes the row (delete).
executions == 0 is "no longer vacuous"	Claim corrected. Mutation: re-execution skips the existing-row branch and enters create. The twin fails on Succeeded (the create resolves no template, a NotFound failure). With that assertion relaxed, the executions count stays 0, because the fixture tags no template release latest and the create stops before the substrate. The doc and table now say the Succeeded check is what catches it.
pod_store_apiserver shortcuts	DELETE refuses a preconditions.uid mismatch with a 409. The Running-on-create, never-scheduled and no-Terminating shortcuts are documented next to the reflector limitation. All 21 twins pass unchanged. Mutation: the substrate sends a wrong precondition UID. Two suspend-path twins now fail (…unless_suspend_claims_it, completed_actions_keep…: pod … changed before suspend retirement); with the old double all 21 passed. live_cpu… passes under it: its retire did not fail on the wrong UID.
Stack headroom	The native-start twin's future is boxed, and both twins carry a one-line margin comment. The two twins changed here box theirs too. Smallest passing RUST_MIN_STACK at CI's opt-level 0, in 128 KiB steps: native-start 1.625 MiB (1.875 before); untracked-actions still 1.875 MiB, since its remaining depth is the shared Kubernetes start path's frames below the boxed future; key-share 1.625 MiB; completed-actions 1.5 MiB.

Mutation check (one per new or changed assertion)

Each mutation was applied to production code (or, where noted, to the fixture), then the twin was run alone under CI's settings. Each file was restored with git checkout. The earlier set is unchanged and was run at the previous head (792e2be); it was not re-run after the rebase. The new set ran at this head:

twin	mutation	result
activations_and_suspend_wait… (missing-pod stop, kill)	lock_sandbox_for_execution_mutation_in FOR UPDATE → FOR NO KEY UPDATE	fails: stop never waited
activations_and_suspend_wait… (missing-pod placement)	fixture: the double binds created pods to the worker node	fails: row on_node, not unplaced
completed_actions_keep… (start after Wake)	drop the terminal guard from maybe_update_sandbox_state	fails: row started; old twin passes
completed_actions_keep… (start after Wake)	drop the terminal guard from maybe_update_sandbox_lifecycle	fails: row started
completed_actions_keep… (start after Wake, placement)	persist_pod_node_placement skips its sandboxes write	fails: row unplaced, not on_node
completed_actions_keep… (suspend at finalization)	maybe_update_sandbox_lifecycle exempts a suspended write from its guard	fails: row suspended; old twin passes
a_contended_delete_precheck…	Kubernetes delete_execution returns SuspendInFlight	fails; old twin passes
a_contended_delete_precheck…	Kubernetes delete_execution returns Failed	passes, by design: the controller deletes the row anyway
admitted_create_re_execution…	re-execution skips the existing-row branch	fails on Succeeded; with that relaxed, the executions check still passes (see above)
pod_store_apiserver DELETE precondition	the substrate sends a wrong preconditions.uid	2 suspend-path twins fail; with the old double all 21 pass

Earlier set, at 792e2be:

twin	mutation	failure
wake_refuses… (entry guard)	wake entry guard admits stopped	wrong refusal for stopped
wake_refuses… (admission)	admission skipped	created gets 500, not 429
wake_refuses… (deferred claim charge)	charge_restore_claim → Ok(())	restoring and suspended each get 503, not 429
wake_refuses… (suspend in flight)	SuspendInFlight maps to 409	409, not 503
wake_refuses… (pod create)	failed pod create → Unavailable	503, not 500
unreachable_runtime…	failed pod create → Unavailable	503, not 500
wake_completion…	drop ensure_pod_exists' lifecycle refresh check	500, not 409
completed_actions_keep…	drop start_execution's recoverable-lifecycle refusal	wrong refusal message
native_start_overwrites…	start writes refuse stopped	sandbox reported Stopped
activations_and_suspend_wait…	materialization lock FOR UPDATE → FOR NO KEY UPDATE	start never waited
stop_and_kill_on_a_draining_node…	drain branch uses the recovering client	refusal arm fails
untracked_…_without_a_row_are_not_found	runtime_client maps NotFound to 503	503, not 404
untracked_…_unless_suspend_claims_it (created)	suspend claim also accepts created	suspend arm fails
untracked_…_unless_suspend_claims_it (started)	suspend claim drops started	503 on the started row
untracked_…_unless_suspend_claims_it (paused)	suspend claim drops paused	503 on the paused row
observed_only_delete…	observed delete skips the node-api call	resident not deleted
redrive…	probe answers NotRoutable for a routable execution	row stays restoring
(redrive, both guards)	probe lies and the CAS drops its routable term	unroutable row completes
exhausting_the_finalize_retry_budget…	exhausted finalize → internal	500
a_contended_delete_precheck…	pre-check error → internal	500
resume_uses_resolved_runtime…	runtime_client skips the pod instance	503
stored_ingress_access_resumes…	ingress skips the paused resume	worker stays paused
postgres_failed_pooled_create_defers…	Kubernetes cleanup returns FinalizeSandboxDbDelete	row deleted
create_executor_retries…	failed pod create → NotFound	not requeued
admitted_create_re_execution…	admitted_pool_cleanup_pending → false	Succeeded, not Queued
untransmitted_update_is_replayed…	probe_runtime_client → None	no fresh client
interrupted_update_is_observed…	RuntimeClient::call replays ambiguous failures	update resent and succeeds
live_cpu… (allocation)	a vCPU update persists the quota	allocation 2, not 8
live_cpu… (next generation)	pod names drop the generation	restarted pod is not -g2
live_cpu… (new pod resolved)	resolution reads the generation-1 pod name	no read of the -g2 pod

Validation

Stack overflow in CI's "common / Rust tests" (fixed in an earlier revision). At 39fe181, characterization_tests::kubernetes::untracked_actions_report_status_and_leave_the_row_unless_suspend_claims_it overflowed its stack in CI. On Linux ci/scripts/run-rust-checks.sh builds pplx-space-api and pplx-space-sbx-mgr at opt-level=0 (TEST_PROFILE_ARGS), not at Cargo.toml's test-profile opt-level=1. Boxing the run_action future fixed it. The four twins that drive the shared start or suspend path now box their action future; the measured margins are in the review follow-up above.

"CI's settings" below means --config profile.test.package.pplx-space-api.opt-level=0 --config profile.test.package.pplx-space-sbx-mgr.opt-level=0, CARGO_PROFILE_DEV_DEBUG=line-tables-only, CARGO_PROFILE_DEV_BUILD_OVERRIDE_DEBUG=false, CARGO_INCREMENTAL=0, CARGO_TARGET_DIR=target/buildkite-tests, NEXTEST_TEST_THREADS=30 and SPACE_CI_REQUIRE_TEST_DEPS=1, on Linux. The nextest setup scripts started Postgres and Redis. With SPACE_CI_REQUIRE_TEST_DEPS=1, a missing dependency fails its test instead of skipping it.

check	result
rebase	onto main 202ef6b, because public_sandbox_operations/tests.rs no longer merged cleanly after #607; the fixture now uses Substrate::from_kubernetes
all 21 twins, CI's settings, after the DELETE-precondition change	21 passed; no outcome changed
cargo nextest run --locked -p pplx-space-api --all-features --retries 0, CI's settings, three consecutive runs at this head	1853 passed, 1 skipped (the #[ignore] benchmark) each run; no overflow
stack, RUST_MIN_STACK in 128 KiB steps, CI's settings	smallest passing: completed-actions 1.5 MiB, key-share 1.625 MiB, native-start 1.625 MiB, untracked-actions 1.875 MiB
cargo fmt --all -- --check	pass
cargo clippy --workspace --locked --all-targets --all-features -- -D warnings at this head	pass, 0 warnings
pre-commit (gitleaks) over the commit	pass
scripts/check_no_inline_rust_test_modules.py	no finding in touched files; the 2 pre-existing findings in crates/libs/observability are unchanged
mutation checks	table above
Kind	not run: tests only

🤖 Generated with Claude Code

[johnlouie-pplx]

Thanks for this, it's a really useful set of twins. The docs on each one make the Direct vs Kubernetes differences easy to follow, and tying the lock waits to pg_blocking_pids plus the sql! locator makes them deterministic. Looks good to merge from my side, with one small non-blocking ask:

Stack headroom. After the Box::pin, the fixed test needs 1.75–1.875 MiB of a 2 MiB test thread stack at CI's opt-level 0, and the description says native_start_overwrites_a_stopped_written_mid_flight_without_a_retry is in the same range. That leaves about 128–256 KiB, so a modest change to the shared Kubernetes start path could overflow one of these and look like an unrelated CI failure. Could you box that twin's action future too, or leave a short comment noting the margin?

[the-gigi-pplx]

Review — 6 findings

Medium

1. crates/libs/api/src/controllers/sandbox/lifecycle/characterization_tests/kubernetes.rs:701 — the key-share finding's "stop, kill: no" holds only when the pod is in the reflector cache, and the twin only seeds that case.
   The PR's own unreachable_runtime… row says wake, stop and kill take the recovering path and recreate a missing pod. That path goes through ensure_pod_exists (pod_ops/ensure_pod.rs:995) to claim_active_execution_pod_materialization (store/pod_placement.rs:109) and then lock_sandbox_for_execution_lifecycle_in, which runs the same SELECT lifecycle FROM sandboxes WHERE id = $1 FOR UPDATE that start and wake block on (store/execution_rows.rs:566). So a stop or kill of a sandbox whose pod is gone waits behind FOR KEY SHARE just like start. Every arm builds its controller from controller_over_pods(.., vec![pod]), so nothing exercises that case. The table in the PR body and the twin's doc present "no" as the action's behavior, and someone fixing the blocking later will read it that way.
   Either scope the claim to "with a live, cached pod", or add a missing-pod arm for stop and kill and pin that they block.
2. crates/libs/api/src/controllers/sandbox/lifecycle/characterization_tests/kubernetes.rs:553 — on Kubernetes, the start and suspend arms of completed_actions_keep_a_terminal_lifecycle_that_landed_mid_flight never reach the guarded post-result write the test is named for.
   The terminal lifecycle is written before run_action, as in the Direct test. Kubernetes re-reads the row before the node call, so start returns its 409 and suspend its 503 before anything runs mid-flight. The passing assertions (sandbox cannot be started while lifecycle is …) show the post-result write is unreachable in those arms. So "row same" for start and suspend holds because nothing wrote the row, and dropping the terminal-state guard from Kubernetes start's lifecycle write wouldn't fail this test. The file already has the hook that does land a write after the node's acknowledgement (install_lifecycle_write_on_route_publication, line 83, used by the wake-completion twin). Using it for start (with a RouteIdentity::Legacy seed) would cover the guard. Suspend needs its own injection point after its claim.

Low

3. …/characterization_tests/kubernetes.rs:719 — "the session's lock_timeout or statement_timeout … can end the waiter". The api sets neither for its sessions, and the application role's lock_timeout is opt-in (SPACE_DB_APP_LOCK_TIMEOUT, docs/configuration.md:424: unset leaves it unchanged). By default, the activation deadline is what bounds a waiter: the start path wraps ensure_pod_exists in its wait budget (adapter/substrate_adapter.rs, ready_wait_bounds). I didn't trace an equivalent bound for suspend's persist_or_accept_suspend_commit wait. Please name the bound that actually applies to each action.
4. crates/libs/api/src/controllers/sandbox/lifecycle/tests/kubernetes.rs:98 — the doc says the retry "reaches the Kubernetes teardown", but the twin keeps the Direct if let Err(e) = &retried { assert!(!e.is_lock_contention()) }. That passes on Ok and on any non-contention error, including one before teardown. The Direct test's reason for the weak form (no substrate to complete against) doesn't apply here, since this twin has a Kubernetes substrate. Assert the retry's actual outcome, or drop the claim.
5. crates/libs/api/src/public_sandbox_operations/tests/kubernetes.rs:141 — the body says the executions == 0 check is "no longer vacuous", but the mutation table, which covers "one per new or changed assertion", has no row for it. The existing-row branch returns before the substrate (public_sandbox_operations.rs:378-395). Whether a regression that re-entered create would leave a sandbox_k8s_executions row, under substrate_with_store's inert client, isn't shown. A mutation that sends re-execution into create would settle it.
6. crates/libs/api/src/test_support/kubernetes.rs:96 — pod_store_apiserver creates pods already Running, and its DELETE removes by name, ignoring the UID and resourceVersion preconditions the substrate sends. A real apiserver keeps the pod Terminating and answers 409 to a mismatched precondition. The live_cpu… twin's retire-then-create of -g2 is "same" partly because of that. Name these two behaviors in the doc next to the reflector limitation, or honor a preconditions.uid mismatch with a 409. That's a few lines, and it makes a wrong-UID regression catchable.

Verified against 792e2be8744aa71b606c14ca9e46189f57bb74fb. All checks are green, including common / Rust tests, which runs these twins. I didn't run cargo or mutations locally. Findings 1 and 2 come from tracing the call chain to the named file:lines, plus, for finding 2, the twin's own passing refusal assertions. I didn't add findings for the redrive arm's double guard, which the body already prices, or for the shared-address simulator limitation, which the body states.

[the-gigi-pplx]

Approved. 6 non-blocking findings in the review comment above.

[ac-pplx]

Thanks both. All seven are addressed in 571ddbb (tests and docs only); the PR body's "Review follow-up" section has the details.

• Key-share, stop/kill with the pod gone: added missing-pod arms; both block, recreate the pod, and end stopped/unplaced. The "no" is now scoped to a cached pod. One correction to the premise: the first wait there is SELECT id … FOR UPDATE (lock_sandbox_for_execution_mutation_in) while resolving the missing pod, not the SELECT lifecycle … FOR UPDATE start blocks on (confirmed with a pg_stat_activity probe).
• Mid-flight terminal twin: start now lands the terminal write after the node's Wake ack via the route-publication hook, and suspend after its own final write via a test trigger; dropping either guard now fails the test. The up-front refusal cases are kept.
• Wait bounds: named per action. Start/wake: the activation wait budget. Stop/kill with the pod gone and suspend's commit: nothing in the api. By default the space_user role's 10 s statement_timeout from bootstrap-roles still ends each of these waits; lock_timeout is opt-in as you said.
• Contended-delete retry: now asserts the retry succeeds and the row is gone.
• executions == 0: the claim was wrong. A re-execution sent into create fails at template resolution before the substrate, so the count stays 0 either way; Succeeded is the check that catches it. Doc corrected.
• pod_store_apiserver: DELETE now answers 409 on a preconditions.uid mismatch, and the doc lists the remaining shortcuts (Running on create, no Terminating phase). All twins pass unchanged; with a wrong UID, two suspend-path twins now fail.
• Stack headroom: boxed the native-start twin's future and the two changed tests, with a margin comment on each. Smallest passing stacks at CI's opt-level 0 are now 1.5–1.625 MiB; the untracked-actions twin still needs ~1.875 MiB because of the shared Kubernetes start path.