Repository navigation
Conversation
Concurrent index builds release the ordinary migration key so a shipped blocking migrator can finish without retaining a snapshot behind the build. Retry from the catalogs and checksum ledger, and acknowledge a WAL fence on recorded reruns. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Send public driver cancellation before aborting the transport, within one absolute cleanup deadline. Separate borrowed migration execution so its operator keeps sole session supervision. Verify backend lock release and preserve interrupted migration errors. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep directory and local cleanup inside the original operation deadline. Borrow migration execution on the retained session, preserve recoverable count changes, and let each session owner send public cancellation before transport abort. Exercise real backend lock release and the public CLI identity gates. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Exercise erasure replay after permanent placement metadata is lost, and normalize the ClickHouse schema checksum read. Add a pending ordinary migration to the physical recovery drill so its schema and ledger commit are interrupted together. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Claim the complete directory registration before schema and metadata repair. Retain its generation through traversal restarts and the final acknowledgement. Publish the raw backend before initialization and keep one independently supervised close inside the original command deadline. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Capture native failures inside owned work and public-close child tasks so their supervisors can clean up before rethrowing. Keep the first migration failure through context exit within its original cleanup deadline. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
A custom context can raise before returning its exit awaitable. Defer that invocation to the owned child so it cannot replace the first migration error. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep each page inside its own statement budget while sharing close, reap and pool-exit supervision with directory controls. Preserve the audited commit and cell-row lock without duplicating a session owner. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> (cherry picked from commit 96df6ab6bc91198f867d0c2acedcb0aca0e044c8)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com> (cherry picked from commit 30a4ac5d4a0f01045db153af727ca11e4ed9066f)
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Remove an obsolete fixture patch for a deleted attach-lock helper so the existing prepare assertions run. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Retain current attachment and placement guards while adding stopped-cell metadata recovery, native selections and generated command references. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
zlareb1
changed the base branch from
feat/cell-attach-cli
to
feat/cell-update
November 18, 2026 13:06
Keep the initiating failure while draining owned scratch tables, clients, tasks and pools within the existing standby-drill cleanup bound. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Reuse the supervised encoding refusal before lock acquisition, and protect test observers before creating restore tasks. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Adopt the provisioning and configuration control dependencies while preserving the stopped-cell recovery protocol and its regression cases. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Bind restore and control owners before deployment and codec initialization can wait. Preserve public close on late connections and initialization failures without changing ordinary dedicated-session cleanup. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Run the shared first-cell write policy on the bound backend under the original command deadline so cancellation closes that session before cancelling its read. Keep paired-store checks and native backend-release coverage. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related: #735
Related: #776
Changelog category (leave one):
Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):
loom cell restorerepairs a stopped cell's tenancy, mode and configuration metadata from the surviving router directory before reconciliation and startup.Why
A restored Postgres backup can lose permanent tenancy rows or retain an older acceptance fingerprint. Reconciliation can then refuse an org whose placement still exists in the router directory. Restore copies the directory's metadata under the cell's retained operator lock and a fresh directory generation claim. It creates no placement and deletes no content.
The command uses bounded ordered pages and acknowledged commits instead of deriving ownership from content or collecting all placements in memory. Rerunning attach requires live probes and cannot replace stopped-cell recovery. A successful repair reports metadata completion separately from prepare, configuration recovery and readiness.
Scope and risk
This draft stacks on the shared cell update and placement control protocol in #776, which includes the SCIM provisioning and attachment foundations. Restore reuses the existing UTF8 refusal before acquiring its control lock. Shared CLI declarations and writer censuses include both commands.
The restore path covers paired-store identity checks, the retained raw Postgres session, pending cell schema, direct probes, tenancy and acceptance repair, strict trigger checks, generation checks, interruption diagnostics and erasure-reconciliation fixtures. The router adds an authenticated, audited page read. Memory use is bounded by page bytes; repeated concurrent placement can exhaust the command deadline.
The cell must remain stopped throughout store recovery, metadata repair and reconciliation. Activity checks can refuse observed live processes but cannot prove every old process has ended. This command does not recover lost erasure evidence, restore revoked credentials, rebuild a lost directory, erase stray orgs or create a separate deployment from a copy. Those obligations remain separate.
Restore, placement and cell control synchronously bind their cleanup owner inside the shared Postgres connector before deployment or codec initialization can wait. A closed owner refuses initialization and publicly closes a late session within its original deadline. Initialization runs once. A binding callback that raises leaves cleanup with the connector; ordinary dedicated callers retain connector cleanup through initialization. Process admission, participant attribution, startup markers and retirement seals still apply. This avoids a second raw connection path or an untracked initialization interval. The separate phase deadlines for placement and control and the single total restore deadline remain intact.
Public preflight verifies the ClickHouse authority, binds the one Postgres backend and runs the shared first-cell write policy on that session before direct probes or repair. The policy keeps its ten-second read-work cap and the command retains one total deadline, including public close. An owned-reader timeout uses a fixed diagnostic; native fatal exceptions remain captured through cleanup, and a policy refusal remains an operator error. The first-cell gate is never downgraded to a read.
The composed parent retains attachment, generation fencing and placement bounds. Required final-head native and full CI gates remain pending; source review and focused checks do not establish the complete recovery workflow.
Compatibility, rollout and rollback
loom cell restore --cell NAME [--router URL] [--wait SECONDS]and router-internalPOST /cells/v1/operator/cells/{cell}/restore-read. No public Operation, SDK field, setting or default changes. Placement and update are supplied by the parent protocol; this diff adds restore and its generated CLI surface.Tests
No broad local suite runs. This draft remains stacked on the update dependency and has no main-branch CI checks yet; zero checks is not a pass. Full unit, coverage, generator, schema, integration, Postgres 16/18, mixed-build and physical interruption/retry/promotion gates must pass on the final main-targeted composition before merge. No latency, ranking-equivalence or scale result is claimed.
AI assistance in the code (leave one; writing this description with AI is fine):
AI-generated (an agent wrote most of the diff): Codex. Checks cover source preservation, scoped formatting and lint, the prepare fixture regression, sign-offs, dependency state and evidence attribution. The earlier independent source reviews report no findings or nits. Fresh independent review of the final pushed thirty-path concern reports no source findings or nits; native and full CI remain required.
I can explain every line of this diff and how it interacts with the rest of Loom.
Checklist
make format-check,make lint,make typecheckandmake test-unitpass; the integration suite passes if server behaviour changed.docs/reference/configuration.mdand thedocs/reference/tables (python scripts/codegen/docs_reference.py), the reach manifest,schemas/schema.sql(make schema-dump) and the Codex plugin copies (python3 scripts/codegen/agent_plugin_sync.py).main; no merged migration is edited.hooks/,lib/,bin/,skills/,.mcp.json, the Codexplugin-defaults.json) bumps both plugin manifests' version, and no other open PR uses that version.npm --prefix frontend run build) and its tests pass, iffrontend/changed.docs/describe the system as it now is, and this description names any operator steps and wire changes.tests/unit/test_public_copy.py).git commit -s), as CONTRIBUTING.md asks.