Skip to content

Stopped cells restore metadata from the router directory - #796

Draft
zlareb1 wants to merge 33 commits into
feat/cell-updatefrom
feat/cell-restore
Draft

zlareb1 wants to merge 33 commits into
feat/cell-updatefrom
feat/cell-restore

Conversation

@zlareb1

@zlareb1 zlareb1 commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

Related: #735
Related: #776

Changelog category (leave one):

  • New Feature

Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):

loom cell restore repairs a stopped cell's tenancy, mode and configuration metadata from the surviving router directory before reconciliation and startup.

Why

A restored Postgres backup can lose permanent tenancy rows or retain an older acceptance fingerprint. Reconciliation can then refuse an org whose placement still exists in the router directory. Restore copies the directory's metadata under the cell's retained operator lock and a fresh directory generation claim. It creates no placement and deletes no content.

The command uses bounded ordered pages and acknowledged commits instead of deriving ownership from content or collecting all placements in memory. Rerunning attach requires live probes and cannot replace stopped-cell recovery. A successful repair reports metadata completion separately from prepare, configuration recovery and readiness.

Scope and risk

This draft stacks on the shared cell update and placement control protocol in #776, which includes the SCIM provisioning and attachment foundations. Restore reuses the existing UTF8 refusal before acquiring its control lock. Shared CLI declarations and writer censuses include both commands.

The restore path covers paired-store identity checks, the retained raw Postgres session, pending cell schema, direct probes, tenancy and acceptance repair, strict trigger checks, generation checks, interruption diagnostics and erasure-reconciliation fixtures. The router adds an authenticated, audited page read. Memory use is bounded by page bytes; repeated concurrent placement can exhaust the command deadline.

The cell must remain stopped throughout store recovery, metadata repair and reconciliation. Activity checks can refuse observed live processes but cannot prove every old process has ended. This command does not recover lost erasure evidence, restore revoked credentials, rebuild a lost directory, erase stray orgs or create a separate deployment from a copy. Those obligations remain separate.

Restore, placement and cell control synchronously bind their cleanup owner inside the shared Postgres connector before deployment or codec initialization can wait. A closed owner refuses initialization and publicly closes a late session within its original deadline. Initialization runs once. A binding callback that raises leaves cleanup with the connector; ordinary dedicated callers retain connector cleanup through initialization. Process admission, participant attribution, startup markers and retirement seals still apply. This avoids a second raw connection path or an untracked initialization interval. The separate phase deadlines for placement and control and the single total restore deadline remain intact.

Public preflight verifies the ClickHouse authority, binds the one Postgres backend and runs the shared first-cell write policy on that session before direct probes or repair. The policy keeps its ten-second read-work cap and the command retains one total deadline, including public close. An owned-reader timeout uses a fixed diagnostic; native fatal exceptions remain captured through cleanup, and a policy refusal remains an operator error. The first-cell gate is never downgraded to a read.

The composed parent retains attachment, generation fencing and placement bounds. Required final-head native and full CI gates remain pending; source review and focused checks do not establish the complete recovery workflow.

Compatibility, rollout and rollback

  • Database prerequisite: restore refuses non-UTF8 databases before control claims or metadata writes. Recreate the database with UTF8 encoding and restore into it before retrying.
  • Wire: adds loom cell restore --cell NAME [--router URL] [--wait SECONDS] and router-internal POST /cells/v1/operator/cells/{cell}/restore-read. No public Operation, SDK field, setting or default changes. Placement and update are supplied by the parent protocol; this diff adds restore and its generated CLI surface.
  • Operator steps: apply the prerequisite router and ClickHouse schema, stop every cell process and writer, restore the store, run metadata restore with the cell configuration and router operator credential, then reconcile before startup. The command applies pending cell Postgres schema only after identity and claim checks. An interrupted or uncertain commit requires repeating the same command before continuing.
  • Rolling upgrades: final mixed-build and synchronous-standby checks remain required. The current update/placement branch and attachment foundation are composed as ancestors. Run all required gates against the final main-targeted head before merging.
  • Rollback: reverting removes the command and router endpoint. Completed metadata repairs and applied additive schema remain. Rollback does not undo permanent placements or restore older acceptance values.

Tests

  • An initial eleven-case regression run fails while exposing unowned initialization, duplicate codecs, missing public close and initialization after a late acquisition. The correction passes 17 directly related cases, including six shared binding, retirement-seal and control-owner controls. Two additional reuse controls pass and refuse acquisition before driver I/O. The regressions use the real connector, initialization and cleanup with driver I/O replaced; they do not establish native backend exit.
  • Thirty-four existing control, placement, restore, fatal-error, late-connection and retirement cases pass in 10.96 seconds. Raw acquisition fakes move to the driver boundary so the real connector performs binding and initialization. All 365 outer assertion ASTs and 23 embedded subprocess assertions in the audited fixture files remain exact. The shared settlement fixture uses the same control-owner connection method as product callers, with every assertion retained.
  • The first correction check records 15 passes and two deadline failures caused by a test waiting for all tasks while holding public close. The test instead resumes at the first completed rendezvous, preserving its deadlines and assertions; the subsequent 17 cases pass in 2.55 seconds. Cancellation payload checks capture the exact exception before Task conversion. Earlier failing evidence remains recorded.
  • Both CLI snapshot generators pass and reproduce their outputs exactly. Five product modules pass scoped mypy checks. All 24 Python files in the current concern pass scoped Ruff lint, formatting and syntax checks; the current and stacked diffs pass whitespace checks. The documentation generator check and the changed architecture page's pinned formatter pass. Earlier formatter and generator admissions refused before execution remain separate from these passing runs.
  • Five public-command regressions fail before the preflight correction and pass afterward. They use real connection initialization, the shared write policy and cleanup with driver I/O replaced, covering short deadlines, ordinary read failure and identical native fatal exceptions after acknowledged close.
  • The finite policy, CLI and census check first records 34 passes and two census failures: native owner connection methods needed explicit callsite classification, and verified connections needed recognition of the existing declared wrapper. Exact native callsites and their positive presence are classified; the original refusal assertions remain. The same 36 cases then pass, including public first/second cancellation, private timeout-text redaction, wrong-cell write/revocation refusal and the retained read-work cap. All 205 audited prior assertion ASTs remain.
  • A native public-command deadline case stalls the real Postgres state read while holding an advisory lock. It checks backend and lock release before close acknowledgement and command return, compares every field in the four cell metadata tables and requires no ClickHouse write. This case is CI-only and unrun. Its final setup-cleanup adjustment follows the focused run without changing product or unit inputs; it has no native validation result.
  • Eight earlier fingerprint, configuration-recovery, repository-reader and control-writer census cases pass on the adopted parent composition. Its 26 restore paths and prior work remain preserved. Existing native initialization/cancellation/error/late-return cases retain all assertions; their raw-driver seam follows the same ownership boundary as the focused regression. Native standby, backend-lock and complete recovery checks remain CI-only and unrun on this correction.
  • On the final parent composition, both actual CLI snapshot checks pass in 0.88 seconds and reproduce both snapshots exactly; the documentation reference check passes with all 59 outputs unchanged. An earlier CLI admission was refused before execution because the shared focused capacity was occupied; it remains incomplete. No unchanged behavioral cohort was rerun solely for the parent advance. The inherited placement durability drill uses the exact participant tag at both waits and preserves all nineteen assertions; its native execution remains pending in CI.
  • Earlier evidence includes encoding refusal, changed-count retry, readiness, second cancellation, erasure-reconciliation and prepare fixtures. Those results remain historical and do not qualify this corrected composition. The earlier 39-path and intermediate 30-path independent reviews report no source findings or nits. The actual parent adoption retains all thirty concern images byte-for-byte, including all existing product and test inputs. Fresh independent review of all thirty paths on the final pushed head reports no source findings or nits; native and full CI verification remain required.

No broad local suite runs. This draft remains stacked on the update dependency and has no main-branch CI checks yet; zero checks is not a pass. Full unit, coverage, generator, schema, integration, Postgres 16/18, mixed-build and physical interruption/retry/promotion gates must pass on the final main-targeted composition before merge. No latency, ranking-equivalence or scale result is claimed.

AI assistance in the code (leave one; writing this description with AI is fine):

  • AI-generated (an agent wrote most of the diff): Codex. Checks cover source preservation, scoped formatting and lint, the prepare fixture regression, sign-offs, dependency state and evidence attribution. The earlier independent source reviews report no findings or nits. Fresh independent review of the final pushed thirty-path concern reports no source findings or nits; native and full CI remain required.

  • I can explain every line of this diff and how it interacts with the rest of Loom.

Checklist

  • make format-check, make lint, make typecheck and make test-unit pass; the integration suite passes if server behaviour changed.
  • Generated files are regenerated in this PR, not by hand: the SDK op tables, docs/reference/configuration.md and the docs/reference/ tables (python scripts/codegen/docs_reference.py), the reach manifest, schemas/schema.sql (make schema-dump) and the Codex plugin copies (python3 scripts/codegen/agent_plugin_sync.py).
  • A new migration is additive and numbered after the latest on main; no merged migration is edited.
  • A behaviour change to any file a plugin ships (hooks/, lib/, bin/, skills/, .mcp.json, the Codex plugin-defaults.json) bumps both plugin manifests' version, and no other open PR uses that version.
  • The frontend builds (npm --prefix frontend run build) and its tests pass, if frontend/ changed.
  • Docs under docs/ describe the system as it now is, and this description names any operator steps and wire changes.
  • Repository text has no dates, people, host names, internal namespaces, ticket or decision numbers; the history goes in this description (tests/unit/test_public_copy.py).
  • Commits are signed off (git commit -s), as CONTRIBUTING.md asks.

Concurrent index builds release the ordinary migration key so a shipped blocking migrator can finish without retaining a snapshot behind the build. Retry from the catalogs and checksum ledger, and acknowledge a WAL fence on recorded reruns.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Send public driver cancellation before aborting the transport, within one absolute cleanup deadline. Separate borrowed migration execution so its operator keeps sole session supervision. Verify backend lock release and preserve interrupted migration errors.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep directory and local cleanup inside the original operation deadline. Borrow migration execution on the retained session, preserve recoverable count changes, and let each session owner send public cancellation before transport abort. Exercise real backend lock release and the public CLI identity gates.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Exercise erasure replay after permanent placement metadata is lost, and normalize the ClickHouse schema checksum read. Add a pending ordinary migration to the physical recovery drill so its schema and ledger commit are interrupted together.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Claim the complete directory registration before schema and metadata repair. Retain its generation through traversal restarts and the final acknowledgement.

Publish the raw backend before initialization and keep one independently supervised close inside the original command deadline.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Capture native failures inside owned work and public-close child tasks so their
supervisors can clean up before rethrowing. Keep the first migration failure
through context exit within its original cleanup deadline.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
A custom context can raise before returning its exit awaitable. Defer that
invocation to the owned child so it cannot replace the first migration error.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep each page inside its own statement budget while sharing close, reap and pool-exit supervision with directory controls. Preserve the audited commit and cell-row lock without duplicating a session owner.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 96df6ab6bc91198f867d0c2acedcb0aca0e044c8)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
(cherry picked from commit 30a4ac5d4a0f01045db153af727ca11e4ed9066f)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Remove an obsolete fixture patch for a deleted attach-lock helper so the existing prepare assertions run.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Retain current attachment and placement guards while adding stopped-cell metadata recovery, native selections and generated command references.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
@zlareb1
zlareb1 changed the base branch from feat/cell-attach-cli to feat/cell-update November 18, 2026 13:06
Keep the initiating failure while draining owned scratch tables, clients, tasks and pools within the existing standby-drill cleanup bound.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Reuse the supervised encoding refusal before lock acquisition, and protect test observers before creating restore tasks.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Adopt the provisioning and configuration control dependencies while preserving the stopped-cell recovery protocol and its regression cases.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Bind restore and control owners before deployment and codec initialization can wait. Preserve public close on late connections and initialization failures without changing ordinary dedicated-session cleanup.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Run the shared first-cell write policy on the bound backend under the original command deadline so cancellation closes that session before cancelling its read. Keep paired-store checks and native backend-release coverage.

Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant