Skip to content

archcheck: count writes of the sandboxes row outside sandbox-store - #690

Merged
ac-pplx merged 1 commit into
mainfrom
ac/sandbox-writes-ratchet
Oct 21, 2026
Merged

ac-pplx merged 1 commit into
mainfrom
ac/sandbox-writes-ratchet

Conversation

@ac-pplx

@ac-pplx ac-pplx commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Phase 0 of giving the control-plane Postgres public.sandboxes row one writer. This PR adds an oracle: an archcheck ratchet that counts, per file, the production code that writes the row outside pplx-space-sandbox-store. Writes of the Postgres sandboxes row move into sandbox-store; this ratchet only shrinks. One commit, configuration only (ARCHITECTURE.toml, architecture-baseline.toml). It makes no production code, schema, wire format, metric or configuration change.

Scope: only the control-plane Postgres public.sandboxes table. sbx-mgr's node-local SQLite database also has a sandboxes table (crates/libs/sbx/**). That is a different database on a different machine, with its own store and fences (sbxmgr_sqlite, dependency_fence.sbxmgr_*). It is not counted, and the rule allows sbx-mgr's crates.

Rebase onto main

The branch was rebased onto 84963a0. Between 59789a1 and there, #734 (a revert of #217) removed the two UPDATE sandboxes SET target_node_id … statements, claim_create_target and release_create_target, from crates/libs/api/src/controllers/substrate/direct.rs. The stale_entries rule is required, so the old entry for that file (7) failed once combined with main.

The counts were refreshed after the rebase onto main, because #734 removed two statements from direct.rs:

Entry Old → new Why
crates/libs/api/src/controllers/substrate/direct.rs 7 → 5 the two target_node_id statements are gone; cargo arch reported exactly this

Totals: 102 → 100 matching lines, 88 → 86 statement heads (56 that write lifecycle, 25 → 23 that write other columns, 7 hard deletes), 14 renderer calls, still 28 files. The inventory below drops those two rows (renumbered) and refreshes the node_drain/store.rs line numbers that moved on main. Every other entry matches its file, and no production file outside allow gained a match. The patterns, allow and modes are unchanged. The review-round sections and the earlier probe rows keep the figures as measured before the rebase.

The rule: [rules.pattern_ratchet.sandbox_writes]

patterns = [
  # H: a statement head on the row's table
  '''(?i)(?:\\[nrt]|\b)(?:UPDATE|DELETE\s+FROM|INSERT\s+INTO|MERGE\s+INTO|TRUNCATE(?:\s+TABLE)?)\s+(?:ONLY\s+)?(?:\\?"?public\\?"?\s*\.\s*)?\\?"?sandboxes\b''',
  # R: the portability SET-list renderer, as a method or path call
  '''(?:\.|::)\s*sandbox_assignments\s*\(''',
  # M: the portability marker-assignment renderers, as calls (not their `fn` definitions or `use` lines)
  '''(?:^|[^\w\s]|\b(?:return|match|if|in|else|while)\s)\s*(?:\w+\s*::\s*)*(?:clear|write|publish_clear_or_preserve)_current_local_artifact_marker_assignments\s*\(''',
]
  • H matches UPDATE, INSERT INTO, DELETE FROM, MERGE INTO and TRUNCATE [TABLE] naming sandboxes. It matches in any case and with any whitespace, with ONLY, a public. schema, or quoting (including an escaped \"sandboxes\" in a Rust string), before any alias. It also matches right after a \n, \r or \t escape in a one-line string ("WITH c AS (…)\nUPDATE sandboxes …"). The trailing \b keeps sandbox_k8s_executions, sandboxes_archive and similar names out.

  • R/M: the statements in suspend.rs, orphaned_sandboxes.rs, started_sandbox_repair/store.rs, failed_restorable_repair/store.rs, published_state.rs and pod_tracking.rs interpolate SET lists from crates/libs/sandbox/portability/src/lib.rs. These are SuspendedK8sFinalizationSql::sandbox_assignments, as a method call or a path call, and the *_current_local_artifact_marker_assignments functions, after punctuation, a line start, or return, match, if, in, else or while. Each call is counted too. So a file that composes UPDATE sandboxes s SET {sandbox_assignments} counts the head and each renderer call. Portability's own definitions, its private sandbox_assignments(...) helper, and use lists don't match, so portability needs no entry or allow. Portability calls that helper bare. A path-qualified call to it there (Self::..., super::...) would match and need a reviewed entry; the comment says so.

  • allow:

    • crates/libs/sandbox/store/**, the owner.
    • crates/libs/sbx/** and crates/apps/sbx/**: sbx-mgr's node SQLite. The comment says it is a different database. The crates are allowed whole. The sbxmgr_sqlite ratchet and the sbxmgr_* dependency fences guard crates/libs/sbx/mgr/** and crates/apps/sbx/**, which today are all of crates/libs/sbx and crates/apps/sbx. A narrower glob would make this required rule fire on moves within sbx-mgr, which another effort is restructuring.
    • The same #[cfg(test)] support and fixture globs as substrate_literals (**/*_tests/**, **/*_test_support.rs, crates/libs/db-test-fixtures/**, …). archcheck's test rule is filename-based, so these count as production. After the rebase, a recount over the same globs finds 32 matching lines in 13 files. #[cfg(test)] code inline in a production file counts as production, so test-only writes go in a *_tests file. I checked each matching file: all are #[cfg(test)] modules or the dev-only fixtures crate.
  • Modes: no_crossing, listed_no_growth and stale_entries are required. no_new_entries and no_raised_caps are advisory, as in kubernetes_access, capability_identity and substrate_literals. On this PR, no_new_entries reports each of the 28 entries, as expected for a new instance.

  • Exception paths (in the comment), each a reviewed baseline edit:

    • a new writer outside the store is a new entry (no_new_entries);
    • another write in a listed file is a raised cap (no_raised_caps);
    • splitting a listed file into modules needs entries for the new files and a lower cap on the old one, because --renames pairs only whole-file renames. node_drain/store.rs, with 15, is a likely candidate.
  • Not caught (the same list as the instance comment):

    • a statement head split across lines (UPDATE and sandboxes on different lines) or built at run time (format!("UPDATE {table} ..."));
    • a COPY sandboxes FROM ...;
    • a statement in a non-.rs file, such as the placement state machine's include_str! claim statements under postgres/sql/;
    • a head right after an escape other than \n, \r or \t (\0, \x0a);
    • a renderer called through an alias or a use ... as rename, passed as a function value (.map(SuspendedK8sFinalizationSql::sandbox_assignments)), or called right after break or yield;
    • database triggers;
    • scripts and smoke harnesses outside workspace members' src/.
  • False positives:

    • the same text in a trailing // or /* */ comment, in #[doc = "…"], in a SQL -- comment, or in an ordinary string such as an error message ("failed to update sandboxes …");
    • another database's statement on a table of the same name outside sbx-mgr's crates.

    Whole-line // comments are not code lines, so the doc comments that mention UPDATE sandboxes don't count. On main, nothing in the tree hits these classes: run over every .rs file outside sbx-mgr, tests included, every match is an upper-case statement head or a renderer call.

Inventory: production writes of Postgres public.sandboxes, refreshed after the rebase onto main (84963a0)

Method: I scanned every .rs and .sql file under crates/ for the three statement heads, across lines and in any case. I also checked for MERGE/TRUNCATE/COPY, for heads built from a {table} placeholder, and for SET-list renderers, and then classified each SET list. Excluded: test files, the #[cfg(test)] support modules above, sbx-mgr's crates, and whole-line comments (four doc and // comments mention a head).

  • Column groups: "placement" means some of placement, active_node_id, target_node_id and placement_changed_at. "Markers" means current_local_artifact_{id,kind,origin}. "Lease" means restore_lease_token and restore_lease_expires_at.
  • "renderer (V)" is portability's sandbox_assignments with target-node mode V. It always writes lifecycle (suspended, or quarantined for quarantine_orphan), placement = 'unplaced', active_node_id = NULL and command_ready_pending = FALSE. Clear adds target_node_id/placement_changed_at = NULL, PreserveOrBackfillActive backfills target_node_id, and the caller's marker assignments are appended.
  • Line numbers are the statement heads, which is where the ratchet's findings point.
# file:line fn columns written guard caller class
1 api/src/controllers/sandbox/lifecycle.rs:180 begin_delete_against_active_fork lifecycle→deleting id only; earlier FOR UPDATE and no-fork check in the same tx API request path
2 api/src/controllers/sandbox/lifecycle.rs:201 restore_lifecycle_after_failed_delete lifecycle→previous lifecycle = 'deleting' API request path
3 api/src/controllers/sandbox/lifecycle.rs:238 restore_lifecycle_after_failed_finalize_delete lifecycle→previous lifecycle = 'deleting' API request path
4 api/src/controllers/sandbox/lifecycle.rs:393 complete_wake_lifecycle lifecycle→started, placement lifecycle NOT IN (terminal/suspend/delete set); suspended only with the fence off (Direct) API request path
5 api/src/controllers/sandbox/queries.rs:59 finalize_created_sandbox lifecycle, placement, active_node_id, placement_changed_at, command_ready_pending, preserve_local_artifact_on_restore_failure creating, or restoring with lease token, or started on the same node create
6 api/src/controllers/sandbox/queries.rs:1465 create_with_sandbox_id_inner DELETE id only (generation reservation failed) create
7 api/src/controllers/sandbox/queries.rs:1542 create_with_sandbox_id_inner DELETE id only (substrate create failed) create
8 api/src/controllers/sandbox/queries.rs:1583 create_with_sandbox_id_inner active_node_id, placement_changed_at id only create
9 api/src/controllers/sandbox/responses.rs:350 maybe_update_sandbox_lifecycle lifecycle→action result lifecycle NOT IN (failed, quarantined, deleted, cleanup_required) API request path
10 api/src/controllers/sandbox/responses.rs:380 maybe_update_sandbox_state lifecycle→started, placement same set API request path
11 api/src/controllers/sandbox/responses.rs:781 update_and_fetch_post_wake_sandbox lifecycle→started, placement, active_node_id, placement_changed_at same set API request path
12 api/src/controllers/sandbox/updates.rs:399 set_network_policy network_policy_override_json, network_policy_generation optional expected generation config verb
13 api/src/controllers/sandbox/updates.rs:520 set_snapshots_enabled_desired snapshots_enabled_desired id only config verb
14 api/src/controllers/sandbox/updates.rs:1054 rollback_created_sandbox DELETE id only create (rollback)
15 api/src/controllers/sandbox/updates.rs:1075 rollback_created_sandbox DELETE id only (fallback when the tx cannot begin) create (rollback)
16 api/src/controllers/sandbox/wake_completion.rs:236 fenced_completion_cas lifecycle→started, placement 'restoring', lease lapsed, routable execution API request path
17 api/src/controllers/substrate/direct.rs:961 persist_owned_restore_route_identity_receipt active_node_id, target_node_id, placement, placement_changed_at restore lease token substrate store
18 api/src/controllers/substrate/direct.rs:2414 claim_direct_restore_lease lifecycle→restoring, lease IN (suspended, restoring), lease lapsed substrate store
19 api/src/controllers/substrate/direct.rs:2480 release_direct_restore_lease lifecycle→suspended (kept on retry), lease 'restoring' and lease token substrate store
20 api/src/controllers/substrate/direct.rs:2530 renew_direct_restore_lease restore_lease_expires_at lease token substrate store
21 api/src/controllers/substrate/direct.rs:2558 complete_direct_restore_lease lifecycle→started, placement lease token only substrate store
22 api/src/controllers/substrate/kubernetes/store/activation_fence.rs:503 clear_command_ready_pending command_ready_pending 'started' and pending substrate store
23 api/src/controllers/substrate/kubernetes/store/execution_rows.rs:660 ensure_active_execution_claim_for_lifecycle restore_lease_expires_at lease token, unexpired substrate store
24 api/src/controllers/substrate/kubernetes/store/pod_placement.rs:213 persist_pod_node_placement active_node_id, target_node_id, placement, placement_changed_at changed values only substrate store
25 api/src/controllers/substrate/kubernetes/store/pod_tracking.rs:431 clear_pod_tracking_in (clock bump) placement, active_node_id, target_node_id, placement_changed_at id only substrate store
26 api/src/controllers/substrate/kubernetes/store/pod_tracking.rs:439 clear_pod_tracking_in (no bump) placement, active_node_id, target_node_id id only substrate store
27 api/src/controllers/substrate/kubernetes/store/pod_tracking.rs:464 recover_markerless_restoring_after_execution_cleanup_in lifecycle→suspended, placement, markers, command_ready_pending, lease 'restoring', no marker, no active execution, safe snapshot repair (request path)
28 api/src/controllers/substrate/kubernetes/store/pod_tracking.rs:588 recover_started_with_current_suspend_snapshot_in lifecycle→suspended, placement, command_ready_pending, markers (renderer call :577) 'started', unplaced, execution proof, eligibility repair (request path)
29 api/src/controllers/substrate/kubernetes/store/restore_lease.rs:251 revert_lifecycle_after_rollback_best_effort lifecycle→suspended NOT IN (failed, quarantined, deleted, cleanup_required), optional token substrate store
30 api/src/controllers/substrate/kubernetes/store/restore_lease.rs:345 begin_restore_lifecycle_for (marker branch) lifecycle→restoring IN (suspended, restoring), or released stopped evacuation; marker set drain (via api)
31 api/src/controllers/substrate/kubernetes/store/restore_lease.rs:397 begin_restore_lifecycle_for (user) lifecycle→restoring, evacuation_restore_target, lease suspended, or markerless restoring with lapsed lease; no live execution substrate store
32 api/src/controllers/substrate/kubernetes/store/restore_lease.rs:520 claim_markerless_restoring_restore_lease lease 'restoring', no marker, lease lapsed, no fresh execution substrate store
33 api/src/controllers/substrate/kubernetes/store/restore_lease.rs:584 release_user_restore_lease restore_lease_expires_at lease token substrate store
34 api/src/controllers/substrate/kubernetes/store/suspend.rs:144 finalize_suspended_sandbox_in renderer (Preserve): lifecycle, placement, active_node_id, command_ready_pending, markers (renderer calls :129, :150) id only; locked SELECT in the same tx substrate store
35 api/src/controllers/substrate/kubernetes/store/suspend.rs:305 persist_or_accept_suspend_commit lifecycle→suspending, markers 'suspending' substrate store
36 api/src/controllers/substrate/kubernetes/store/suspend.rs:339 claim_suspend_execution lifecycle→suspending expected lifecycle and running execution substrate store
37 api/src/controllers/substrate/kubernetes/store/suspend.rs:679 finalize_missing_suspending_pod_with_current_snapshot renderer (Preserve): lifecycle, placement, active_node_id, command_ready_pending, markers (renderer calls :671, :678) id only; locked SELECT 'suspending' in the same tx substrate store
38 api/src/controllers/substrate/mod.rs:357 persist_optional_current_local_artifact markers id only substrate store
39 placement/machine/src/postgres.rs:105 completion_sql (complete_placement) lifecycle→COALESCE(marker, started), placement, evacuation_restore_target, lease fence (lease + generation) or lapsed restoring; marker not 'stopped' no caller
40 sandbox/teardown/src/lib.rs:129 finalize_sandbox_db_delete_tx DELETE id only; guarded by its callers delete
41 cluster/controller/server/src/drained_node_terminator/store.rs:275 mark_unrecoverable_sandboxes_lost_query lifecycle→failed 'suspended', terminated target, not durable, settle guard drain
42 cluster/controller/server/src/drained_node_terminator/store.rs:647 mark_unrecoverable_sandboxes_lost placement, active_node_id id = ANY (rows the previous statement failed) drain
43 cluster/controller/server/src/failed_restorable_repair/store.rs:145 revert_failed_to_suspended renderer (Clear): lifecycle, placement, active_node_id, command_ready_pending, target_node_id, placement_changed_at, markers, lease (renderer calls :132, :155) 'failed', aged, lease lapsed, no execution repair
44 cluster/controller/server/src/node_drain/store.rs:125 claim_evacuation_suspend lifecycle→suspending, evacuation_restore_target expected lifecycle, or own marked suspending, on a force-draining node drain
45 cluster/controller/server/src/node_drain/store.rs:360 claim_stopped_evacuation evacuation_restore_target→stopped 'stopped' on the node, marker and pin checks drain
46 cluster/controller/server/src/node_drain/store.rs:501 record_stopped_evacuation_artifact markers node, lifecycle, marker, pin and drain-phase checks drain
47 cluster/controller/server/src/node_drain/store.rs:583 revert_evacuation_suspend_claim lifecycle→previous, evacuation_restore_target 'suspending', no suspended execution drain
48 cluster/controller/server/src/node_drain/store.rs:1253 schedule_restore_retry_query restore_attempt_count, next_restore_attempt_at marker set drain
49 cluster/controller/server/src/node_drain/store.rs:1284 redrive_stranded_suspending_query lifecycle→suspended 'suspending', marker, unplaced, artifact drain
50 cluster/controller/server/src/node_drain/store.rs:1485 mark_sandbox_lost_query lifecycle→failed 'suspended', target node, not durable, no upload drain
51 cluster/controller/server/src/node_drain/store.rs:1525 mark_sandbox_lost_on_gone_node_query lifecycle→failed 'suspended', gone node, no ready snapshot, identity guards drain
52 cluster/controller/server/src/node_drain/store.rs:1682 adopt_remote_snapshot_on_gone_node_query markers, evacuation_restore_target, placement_changed_at 'suspended', target node, aged, ready snapshot, identity guards drain
53 cluster/controller/server/src/node_drain/store.rs:1753 mark_sandbox_failed_on_node_query lifecycle→failed, evacuation_restore_target lifecycle in set, on node, SKIP LOCKED drain
54 cluster/controller/server/src/node_drain/store.rs:1830 adopt_remote_snapshot_for_drain_query markers, evacuation_restore_target, placement_changed_at 'suspended', target node, aged, ready snapshot drain
55 cluster/controller/server/src/node_drain/store.rs:1915 abandon_stale_resident_sandboxes_query lifecycle→failed lifecycle in set, aged, node draining drain
56 cluster/controller/server/src/node_drain/store.rs:2010 fail_resident_sandboxes_on_gone_node_query lifecycle→failed, evacuation_restore_target lifecycle in set, draining node, uid match drain
57 cluster/controller/server/src/node_drain/store.rs:3406 SubstrateSink::restore_evacuated_and_finalize lifecycle→target, placement, evacuation_restore_target 'restoring' drain
58 cluster/controller/server/src/node_drain/store.rs:4499 SubstrateSink::evacuate_sandbox lifecycle→suspended 'suspending' drain
59 cluster/controller/server/src/orphaned_sandboxes.rs:1602 recover_orphan_to_suspended renderer (Clear): lifecycle, placement, active_node_id, command_ready_pending, target_node_id, placement_changed_at, markers (renderer calls :1575, :1582) lifecycle <> ALL, classify guard, eligible repair
60 cluster/controller/server/src/orphaned_sandboxes.rs:1663 quarantine_orphan renderer (Clear, quarantined): lifecycle, placement, active_node_id, command_ready_pending, target_node_id, placement_changed_at (renderer call :1644) same, not eligible repair
61 cluster/controller/server/src/orphaned_sandboxes/quarantine_recovery/store.rs:139 recover lifecycle→suspended, placement, markers, command_ready_pending, lease, lifecycle_changed_at, updated_at id only; candidate re-read under locks repair
62 cluster/controller/server/src/started_sandbox_repair/store.rs:206 revert_with_execution_identity renderer (Clear): lifecycle, placement, active_node_id, command_ready_pending, target_node_id, placement_changed_at, markers (renderer calls :192, :217) 'started', aged, no execution, max generation, eligible repair
63 cluster/controller/server/src/suspended_execution_reaper/store.rs:206 retire_stale_execution placement, target_node_id, active_node_id, markers markerless current snapshot predicate sweeper
64 cluster/controller/server/src/tombstone_retention.rs:168 prune_tombstones DELETE 'deleted' rows past retention sweeper
65 cluster/controller/server/src/transient_lifecycle_sweeper/store.rs:75 HEAL_COMMAND_READY_PENDING_SQL command_ready_pending managed_by, pending sweeper
66 cluster/controller/server/src/transient_lifecycle_sweeper/store.rs:118 REPAIR_FAILED_ACTIVE_EXECUTION_SQL lifecycle→started 'failed', on node, active execution, Kubernetes sweeper
67 cluster/controller/server/src/transient_lifecycle_sweeper/store.rs:140 REAP_COMMAND_READY_PENDING_SQL lifecycle→failed 'started' and pending sweeper
68 cluster/controller/server/src/transient_lifecycle_sweeper/store.rs:180 undo_revert_to_suspended_query lifecycle, preserve_local_artifact_on_restore_failure 'suspended' and own write token sweeper
69 cluster/controller/server/src/transient_lifecycle_sweeper/store.rs:196 undo_reap_command_ready_pending_query lifecycle→started, command_ready_pending 'failed' and own token sweeper
70 cluster/controller/server/src/transient_lifecycle_sweeper/store.rs:519 fail_candidate_query lifecycle→failed expected transient lifecycle, evacuation guards, age sweeper
71 cluster/controller/server/src/transient_lifecycle_sweeper/store.rs:532 undo_fail_aged_candidate_query lifecycle 'failed' and own token sweeper
72 cluster/controller/server/src/transient_lifecycle_sweeper/store.rs:569 converge_restoring_query lifecycle→started/paused 'restoring', aged, lease lapsed, routable sweeper
73 cluster/controller/server/src/transient_lifecycle_sweeper/store.rs:609 revert_to_suspended_query lifecycle→suspended 'restoring', aged, ready snapshot sweeper
74 cluster/controller/server/src/transient_lifecycle_sweeper/store.rs:722 fail_unrestorable_restoring_query lifecycle→failed 'restoring', unplaced, no source or upload sweeper
75 internal/api/server/src/data_plane_sessions/service_route.rs:333 clear_sandbox_command_ready_pending command_ready_pending 'started' and pending API request path (service route)
76 internal/api/server/src/published_state.rs:1780 DELETE_ABSENT_STOPPED_SANDBOXES DELETE Direct, stopped, absent from census, not durable ingest
77 internal/api/server/src/published_state.rs:2832 publish_sandbox_states (activity) last_access_at, last_command_at, last_used_at published row; only forward by more than a minute ingest
78 internal/api/server/src/published_state.rs:2916 publish_sandbox_states lifecycle→started 'restoring', node says started, routable, no marker ingest
79 internal/api/server/src/published_state.rs:3016 publish_sandbox_states lifecycle→stopped IN (started, paused, suspended, stopping, restoring) ingest
80 internal/api/server/src/published_state.rs:3144 publish_sandbox_states lifecycle→quarantined claimed from-lifecycle ingest
81 internal/api/server/src/published_state.rs:3460 project_node_published_failed_sandboxes lifecycle→failed NOT IN (failed, quarantined, deleted, cleanup_required, deleting) ingest
82 internal/api/server/src/published_state.rs:3747 apply_suspended_k8s_finalization renderer (PreserveOrBackfillActive): lifecycle, placement, active_node_id, command_ready_pending, target_node_id, markers (renderer calls :3622, :3754) IN (started, paused, restoring) or committed suspending ingest
83 internal/api/server/src/sandbox_quarantine_store.rs:387 consume_activation_completion_receipts lifecycle→receipt target, placement NOT IN (suspending, suspended, quarantined, deleting, deleted, cleanup_required) ingest
84 internal/api/server/src/sandbox_quarantine_store.rs:558 apply_activation_rollback_candidates lifecycle, placement, command_ready_pending, lifecycle_changed_at NOT IN (quarantined, deleted, deleting, cleanup_required) ingest
85 internal/api/server/src/sandbox_quarantine_store.rs:645 recover_pause_failed_from_inventory lifecycle→started/paused 'quarantined' and matching hold ingest
86 internal/api/server/src/status.rs:44 PRUNE_ABSENT_NODE_SANDBOXES_QUERY (prune_absent_node_sandboxes) lifecycle→deleted, placement, command_ready_pending, lease, lifecycle_changed_at, updated_at NOT IN (deleted, deleting), Direct, absent from census, not durable stopped ingest

The table has 86 statements: 56 write lifecycle, 23 write other columns of the row (placement, lease, markers, evacuation_restore_target and restore backoff, command_ready_pending, activity timestamps, network policy, snapshots_enabled_desired), and 7 are hard deletes. The 14 renderer calls are at suspend.rs :129, :150, :671, :678; pod_tracking.rs :577; orphaned_sandboxes.rs :1575, :1582, :1644; started_sandbox_repair/store.rs :192, :217; failed_restorable_repair/store.rs :132, :155; and published_state.rs :3622, :3754.

Writes the ratchet does not count

Where What Why
crates/libs/sandbox/store/src/create.rs:43, :98 insert_unplaced, insert_pooled (INSERT INTO sandboxes) the owner (allow)
crates/libs/sandbox/store/src/lib.rs:474, :515 update_desired_state (metadata, vcpu, memory_mib), set_auto_lifecycle_policy (auto_lifecycle_json, auto_lifecycle_state_json, next_auto_event_at) the owner (allow)
crates/libs/placement/machine/src/postgres/sql/user_restore_claim.sql:44 claim_placement(UserRestore): lifecycle→restoring, evacuation_restore_target, lease; guard: suspended or lapsed restoring, no live execution; no production caller .sql loaded with include_str!; archcheck reads only .rs
…/sql/evacuation_restore_claim.sql:49 claim_placement(Evacuation): lifecycle→restoring, lease; guard: marker set, lease and backoff lapsed; no production caller same
scripts/admin/prune_cluster_artifacts.py, scripts/repro-same-node-restore-no-remote-fallback.py, space_smoke/adapters.py, scripts/tests/* operator script, repro script, smoke harness and script tests outside workspace members' src/
sbx-mgr (crates/libs/sbx/**) node-local SQLite sandboxes different database (allow)
#[cfg(test)] support and fixtures 16 fixture writes test code (allow)

Totals (86 counted statements; 100 matching lines with the renderer calls)

Crate Statements Matching lines (baseline)
api 38 43
cluster-controller 34 41
internal-api 12 14
placement 1 1
sandbox-teardown 1 1
total 86 100 (28 files)
Caller class Statements
substrate store 19
drain (one reached through the api's restore lease) 18
sweeper 12
ingest 11
API request path (one internal-api service route) 9
create (4 create, 2 rollback) 6
repair (two run inside the request path) 7
config verb 2
delete (the shared teardown, see below) 1
no production caller 1
  • finalize_sandbox_db_delete_tx (sandbox-teardown) is called from API delete and the public-operation executor, API create rollback, internal-api's auto-lifecycle delete receipts, cluster-controller's orphan classification and warm-pool cleanup.
  • Several substrate-store writers are also reached from cluster-controller: drain through SubstrateSink, and pod cleanup. The API writers also run inside cluster-controller's public operation dispatcher.

Probe

Per pattern (each probe line was checked alone against each pattern with rg, which uses archcheck's regex engine): 45 positives each matched only their own pattern (H 26, R 6, M 13); 36 look-alikes (H 21, R 8, M 7) and 8 listed misses matched none; the 5 listed false positives matched. Lines added for the review round are in bold.

Pattern Positives (all matched) Look-alikes (none matched)
H UPDATE sandboxes SET …; UPDATE sandboxes s SET; UPDATE sandboxes AS s; update sandboxes set; UPDATE sandboxes<TAB>SET; UPDATE public.sandboxes; UPDATE \"sandboxes\"; r#"UPDATE "public"."sandboxes""#; UPDATE ONLY sandboxes; bare "UPDATE sandboxes"; format!("UPDATE sandboxes s SET {sandbox_assignments} …"); WITH moved AS (UPDATE sandboxes …); sql!("DELETE FROM sandboxes …"); DELETE FROM sandboxes s; delete from sandboxes; DELETE FROM public.sandboxes; INSERT INTO sandboxes (…); INSERT INTO public.sandboxes (; insert into sandboxes(id); MERGE INTO sandboxes s USING …; TRUNCATE sandboxes; TRUNCATE TABLE sandboxes CASCADE; "WITH c AS (…)\nUPDATE sandboxes s SET …"; "SELECT pg_advisory_xact_lock($1);\tDELETE FROM sandboxes …"; "SELECT 1;\r\nINSERT INTO sandboxes …"; "\rUPDATE sandboxes …" UPDATE sandbox_k8s_executions e; UPDATE sandbox_port_exposures spe; DELETE FROM sandbox_quarantine_state; DELETE FROM sandbox_direct_route_identities; INSERT INTO sandbox_events; INSERT INTO sandboxes_history; UPDATE sandboxes_archive; SELECT … FROM sandboxes … FOR UPDATE; FOR NO KEY UPDATE OF s; FOR UPDATE OF sandboxes; ON CONFLICT … DO UPDATE SET; UPDATE nodes; UPDATE sandbox_k8s_executions … FROM sandboxes s; SELECT lifecycle FROM sandboxes; "updated sandboxes after drain"; let sandboxes_updated = …; "… WHERE id = $1\nFOR UPDATE"; "WITH c AS (SELECT 1)\nUPDATE sandbox_k8s_executions e …"; "SELECT 1;\tDELETE FROM sandboxes_archive …"; "rows:\n{n} updated sandboxes"; "COPY sandboxes TO STDOUT"
R sandbox_assignments = finalization.sandbox_assignments(),; let a = finalization.sandbox_assignments();; chained .sandbox_assignments();; suspended_k8s_finalization_sql(…).sandbox_assignments(); SuspendedK8sFinalizationSql::sandbox_assignments(&finalization); pplx_space_sandbox_portability::SuspendedK8sFinalizationSql::sandbox_assignments(&f) SET {sandbox_assignments}; pub fn sandbox_assignments(&self); fn sandbox_assignments(; private call sandbox_assignments( at line start; .active_execution_assignments("e"); .open_port_exposure_assignments(); quarantine_state_conflict_assignments(); render_sandbox_assignments(&f)
M let m = clear_…( ; line-start clear_…(; write_…(; line-start publish_clear_or_preserve_…(; pplx_space_sandbox_portability::clear_…(; render(&clear_…(; format!("SET {}", write_…(…)); return clear_…(…);; match write_…(…) {; if publish_clear_or_preserve_…(…).is_empty() {; for part in clear_…(…).split(',') {; } else if write_…(…).is_empty() {; while pplx_space_sandbox_portability::clear_…(…).is_empty() { the three pub fn …( definitions; pub(crate) fn write_…(; SuspendedTargetNodeAssignment, clear_…, (use list); use …::write_…;; matched_clear_…("s") (a longer name)
  • Listed misses (none matched): "UPDATE / sandboxes SET … split over two lines; format!("UPDATE {table} SET …"); format!("{verb} sandboxes SET …"); "COPY sandboxes FROM STDIN"; use …::clear_current_local_artifact_marker_assignments as clear_marker; and then clear_marker(…); include_str!("postgres/sql/user_restore_claim.sql").
  • Listed false positives (all matched): a trailing // … UPDATE sandboxes SET …; #[doc = "UPDATE sandboxes …"]; a SQL -- DELETE FROM sandboxes … comment; anyhow!("failed to update sandboxes on node {node}"); an sbx-mgr-style "UPDATE sandboxes SET state = ?1 WHERE id = ?2" outside sbx-mgr's crates.

Through archcheck (scratch files, deleted afterwards; the rows up to the probe rows were measured before the rebase, the last two after it):

Tree sandbox_writes result
main before the rebase (59789a1) + this ARCHITECTURE.toml, base baseline (no table) 28 required no_crossing findings, 102 lines; cargo arch exits 1
origin/main at 188c1a3 (before #734) + this config and baseline 0 required; the same 28 files and 102 lines (no cap changes)
this head before the rebase (f03301b) no sandbox_writes finding without --base-baseline; with the CI flags, 28 advisory no_new_entries (one per entry); cargo arch exits 0
head + a scratch production file holding all probe lines (before the rebase) 1 required no_crossing: exactly 50 lines (45 positives + 5 false positives)
rebased head (84963a0 + this commit), before the fix 1 required stale_entries (direct.rs 7 → 5); cargo arch exit 1
rebased head with direct.rs lowered to 5 no sandbox_writes finding; with the base comparisons (--base-baseline, --base-architecture) 28 advisory no_new_entries, no no_raised_caps; cargo arch exits 0
head + four real sbx-mgr SQLite statements in crates/libs/sbx/mgr/store/src/db/ and in crates/apps/sbx/mgr/cli/src/ not counted (the CLI copy is caught by sbxmgr_sqlite, as it should be)
head + one Postgres write each in a *_test_support.rs, a *_tests/ module and crates/libs/sandbox/store/src/ not counted
head + one more DELETE FROM sandboxes in tombstone_retention.rs required listed_no_growth (1 → 2)
head with one updates.rs head renamed off the table required stale_entries (4 → 3)

The rule matches exactly the inventory above, per file. It matches no sbx-mgr statement, comment, doc string or test.

Review round 1

Finding Change
MINOR: a head right after a \n/\r/\t escape in a one-line string was missed H starts with `(?i)(?:\[nrt]
MINOR: a marker-renderer call after a keyword was missed M's prefix adds `\b(?:return
NIT: complete "Not caught" R now also matches a path call (`(?:.
NIT: document the exception paths the comment and this body list new entry, raised cap, module split (new entries plus a lower cap; --renames pairs only whole-file renames) and inline #[cfg(test)] writes (move them to a *_tests file)
Keep the broad sbx-mgr allow kept; the comment now says the sbxmgr_* ratchet and fences already guard those crates, and a narrower glob would fire on moves within sbx-mgr

None of the newly caught forms occur in the tree, so the baseline is unchanged: 28 files, 102 lines, at this branch's base and at the then-current origin/main. The branch merged cleanly at that point, so it was not rebased yet.

Review round 2

Comment wording only; the patterns and baseline are unchanged.

Finding Change
The sbx-mgr allow wording overstated what the sbxmgr_* rules guard The comment now says those rules guard crates/libs/sbx/mgr/** and crates/apps/sbx/**, which today are all of crates/libs/sbx and crates/apps/sbx. The glob is kept.
R's :: form would catch a path-qualified call to portability's private helper One sentence: portability calls it bare, and a Self::/super:: call there needs a reviewed entry. Checked: both forms match, the bare call doesn't.
"Not caught" was missing three forms Added a renderer passed as a function value, break/yield-prefixed calls, and heads after a \0/\x0a escape. Each was checked to miss. This body's list matches the comment.

Validation

Check Result
cargo arch (rebased) exit 0; 98 advisory findings, 0 required, the same 98 as main (84963a0)
cargo arch --format rdjson --base-baseline <main> --renames <diff> --base-architecture <main> exit 0; 126 findings = main's 98 + 28 pattern_ratchet.sandbox_writes.no_new_entries warnings; 0 ERROR
cargo test --locked -p pplx-space-archcheck (unit, tests/cli.rs, tests/repository.rs) pass (7 + 47 + 5)
Probe (above) as expected, 0 unexpected lines
pre-commit (Gitleaks) pass
This PR's CI all 21 checks that ran pass on 4390797 (including rust_static / Rust clippy, which runs cargo arch with the base comparisons, and common / Rust tests); kind_smoke is skipped

No Rust code changes, so no workspace build or test beyond archcheck's own; per the repository's validation rules, a configuration-only change needs only the checks that prove it.

Notes for review

  • No crate-doc edit. I didn't add a pointer to this rule in the store's crate docs (crates/libs/sandbox/store/src/lib.rs). sql! locators embed line!(), so any line added there would change the statement text of every verb below it. The rule's comment in ARCHITECTURE.toml and the baseline header carry the contract instead.
  • Docs that later phases must reconcile. The store's crate docs and the space-data-store-development skill still say that guarded transitions are not store verbs and that placement transitions belong to pplx-space-placement. This ratchet counts both as writes outside the store, which is the plan's direction. Updating that text belongs with the phase that moves the first guarded transition, so this PR doesn't touch it.
  • The store's crate docs (crates/libs/sandbox/store/src/lib.rs) say there are "81 production UPDATE sandboxes statements" outside the node-local table. That figure predates Revert "Preserve authenticated scope in cached template lookups (#217)" #734, which removed two of them (see "Rebase onto main" above); this inventory has two fewer: 79 UPDATE heads in .rs files outside the store (86 − 7 deletes).

🤖 Generated with Claude Code

@ac-pplx
ac-pplx force-pushed the ac/sandbox-writes-ratchet branch 2 times, most recently from 2c67df9 to f03301b Compare October 20, 2026 06:03
@ac-pplx
ac-pplx marked this pull request as ready for review October 20, 2026 06:24
@ac-pplx
ac-pplx added this pull request to the merge queue Oct 21, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 21, 2026
Writes of the control-plane Postgres `public.sandboxes` row are to move
into its store, `pplx-space-sandbox-store`. A new archcheck instance,
`pattern_ratchet.sandbox_writes`, counts the writers that are still
outside it, per file, so the count can only shrink. Configuration only:
no production code, schema, wire format, metric or configuration change.

- ARCHITECTURE.toml: the instance counts each production line that opens
  a statement on the row's table (`UPDATE`, `INSERT INTO`, `DELETE FROM`,
  `MERGE INTO` or `TRUNCATE` naming `sandboxes`; any case, extra
  whitespace, `ONLY`, a `public.` schema or quoting, before any alias,
  also right after a `\n`, `\r` or `\t` escape), and each call to the
  sandbox-portability renderers that build part of its SET list
  (`sandbox_assignments()` as a method or path call, and the three
  `*_current_local_artifact_marker_assignments` functions, also after
  `return`, `match`, `if`, `in`, `else` or `while`). `allow` covers the
  store, sbx-mgr's crates whole (node-local SQLite has its own
  `sandboxes` table, a different database already guarded by the
  `sbxmgr_*` rules, and a narrower glob would fire on moves within
  sbx-mgr), and the `#[cfg(test)]` support modules and fixtures that
  archcheck's filename-based test rule counts as production, the same
  list `substrate_literals` uses. `no_crossing`, `listed_no_growth` and
  `stale_entries` are required; `no_new_entries` and `no_raised_caps` are
  advisory. The comment lists what the patterns miss, their false
  positives, and the exception paths: a new entry for a new writer, a
  raised cap for another write in a listed file, new entries and a lower
  cap for a module split, and a `*_tests` file for test-only writes.
- architecture-baseline.toml: `[pattern_ratchet.sandbox_writes]` lists
  the 28 files that write the row today, 100 matching lines in all: 86
  statement heads (56 that write `lifecycle`, 23 that write other columns
  of the row, 7 hard deletes) and 14 renderer calls. Other tables are
  unchanged.

The two placement-machine claim statements under
`crates/libs/placement/machine/src/postgres/sql/` are loaded with
`include_str!`, and archcheck reads only `.rs` files, so they are not
counted.
@ac-pplx
ac-pplx force-pushed the ac/sandbox-writes-ratchet branch from f03301b to 4390797 Compare October 21, 2026 04:19
@ac-pplx
ac-pplx added this pull request to the merge queue Oct 21, 2026
Merged via the queue into main with commit b9668ad Oct 21, 2026
22 checks passed
@ac-pplx
ac-pplx deleted the ac/sandbox-writes-ratchet branch October 21, 2026 04:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants