Repository navigation
Conversation
…are takes the same lock The attach command holds the attach lock on a direct, marked session of the cell's Postgres for its run and resumes by the directory row and the cell's mode. It refuses before any write on the stores' identity, the session, the roles, settings and subscriptions that would let an unchecked writer through, the database encoding, durability, its own grants, the ClickHouse writers, the table list and the processes' heartbeats; posts attach/begin; creates the triggers under the migration lock; in recording scans and writes the tenancy rows, refusing an org past ORG_MAX_BYTES; commits enforced, proves it durable, waits out every transaction that could have read recording, rescans, places every org in batches and marks the cell serving. A refusal after the mode commit leaves the cell enforced and attaching, and a rerun resumes there. Prepare refuses a session behind a pooler or a database that is not UTF8, and takes the attach lock. Doctor runs attach's Postgres reads again and reads stray orgs with --stray-orgs. instances.read_recent keeps stopped rows inside the widened window, and PRUNE_AFTER_S names the sweep's hour. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…t gate measured end to end docs/operations/cell-attach.md is the operator's procedure: the preconditions attach checks and those it relies on, the flags, the point of no return at the mode commit, the orgs attach does not place and the manual removal of an org past the length limit, and doctor's reads after serving. cells.md describes the first attach as it runs (the lock, the refusals, the trigger pass, the mode commit and its durable read, step 4's wait, rescan and final checks), with its stated limits and failure rows. upgrades.md step 18 rotates the ClickHouse writer credential after the LOOM_CELL rollout. bench/cell_trigger_gate.py measures memory.set and a delivery-sized insert through an attached cell with the org triggers on and off; results.md records a local run. Attach compares the final root-secret fingerprint with the one its begin wrote, and the nonce cleanup's message names no single command. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…s no boot or insights module instances.read_recent takes the window it reads back; attach's heartbeat module computes it from the live window and a stopping process's drain and retirement budgets. PRUNE_AFTER_S is marked as the contract it is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… in wall time, and says when the mode commit's outcome is unknown The refusals name every table the trigger pass would alter whose owner role attach's role is not a member of, before any write; ensure_table and _run_ddl answer applied, busy or denied, so a denial during the pass refuses at once rather than after --wait as a busy lock. --wait is wall time: each bounded wait charges its elapsed time, lock tries included, and every lock try waits at most what is left; scan time stays outside it. From the mode commit on, a stop that comes before the durable read says the commit's outcome is unknown and to rerun attach, for a refusal and for any other error alike. The prune-window check runs with the refusals, before any write; an enforced resume runs one trigger pass; the lock session opens inside the block that closes it; the durable read retries a timeout or an outage only; refusals name no command that does not exist. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… the attach docs describe one step numbering and today's procedure --stray-orgs labels only a spent budget as tables left unchecked; any other refusal names its table and the read continues. A failed catalog read reports both of attach's Postgres checks as not read. The runbook and doctor describe removing a stray or over-long org by hand, and name no command that does not exist yet. cells.md labels the first attach's list with the design's step names, so step 1 to step 4 mean the same everywhere, rewrites the stuck-process failure row and the planned tests for a mode that only rises, and says --wait is wall time. The trigger-cost harness uses cell_triggers.create_statement and deletes the rows its inserts wrote. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… inside --wait, and doctor warns on the role it runs as The mode is taken from the first state read, set to unknown only once the router accepted attach/enforced, and set from what the durable read saw; an interrupt prints the mode line and goes on. The trigger DDL's lock bound is taken after the migration lock, from what --wait has left then. An ended mode transaction whose lock does not clear refuses, naming its backend. A busy table's holders come back with ensure_table's answer, and attach says them once per change rather than a boot's warning a try. Attach counts the role it runs as among its own; doctor does not. The dead SCIM recheck and the unused scim_ok parameter are gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
… figures say what they establish cells.md and the runbook state that attach exempts the role it runs as and doctor does not. results.md gives the marked batch's per-round changes and their mean, says the delivery-pass gate is met on the mean but not established by three rounds alone, and that the memory.set p99 gate is unresolved locally. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…nknown only while a try's UPDATE is out, and a busy migration lock is named Each trigger and mode-commit try gets a lock wait of at least TRY_FLOOR_MS before --wait is judged spent, so --wait 0 is one try of each wait with no retry, and an idle cell attaches with it. The run's mode turns unknown just before a try sends its UPDATE and back to recording when that try rolls back. A trigger try that finds the migration lock busy names that lock's holder, in the retry line and in the refusal. The step-4 watch task is awaited after its cancel. The ATTACH_LOCK_KEY comment says who holds the lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
…span on one line inside its list items Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Check each probe privilege separately because a comma-separated privilege list accepts any one of them. Run the attach integration tests in the Postgres matrix and limit their heartbeat reads to the test's own process IDs while retaining the real queries. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Manage mode and direct-session transactions explicitly so a canceled statement cannot start an unbounded rollback. Roll back ordinary mode statement errors within the try's deadline; terminate synchronously when a try ends with an error or cancellation. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Give each trigger attempt a client deadline covering migration lock acquisition, DDL, commit, rollback and unlock. Use explicit transactions so cancellation terminates the session without waiting for rollback or masking the original error with lock cleanup. Give gate batch scopes a unique prefix per run so cleanup preserves existing rows. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Count privileges a login can reach through SET ROLE or an ADMIN regrant, including noninherited group grants. Keep the owner exemptions and explicit role allow list. Repeat the writer, trigger, Postgres catalog and standby checks after the rescan and placement before marking the cell serving. Retain the heartbeat watch and its final wait floor. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Check USAGE of pg_read_all_stats and pg_signal_backend because attach uses their privileges without SET ROLE. Keep MEMBER for the separate untrusted-writer reachability checks. Refuse noninherited capabilities before attach writes, and explain the inherited grant requirement in the refusal labels and runbook. Test both inherited neighbors and signaling a scratch backend. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Concurrent index builds release the ordinary migration key so a shipped blocking migrator can finish without retaining a snapshot behind the build. Retry from the catalogs and checksum ledger, and acknowledge a WAL fence on recorded reruns. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Claim a fresh directory UUID before acceptance writes, and compare the generation, registration and exact fingerprint pair on control mutations. Keep every local acceptance transaction on its attach-lock backend and stop after a lost session. The additive router column permits older data-plane reads and inserts. Online control requires turnover of older clients and router handlers; the recovery guide covers claimed state cleanup and offline generation rotation. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Refuse explicit open requests under the guarded attaching row lock, preserving quota atomicity and the durable zero-placement boundary. Inventory control writers and their required boundaries, including dynamic cleanup SQL. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Concurrent index builds release the ordinary migration key so a shipped blocking migrator can finish without retaining a snapshot behind the build. Retry from the catalogs and checksum ledger, and acknowledge a WAL fence on recorded reruns. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Send public driver cancellation before aborting the transport, within one absolute cleanup deadline. Separate borrowed migration execution so its operator keeps sole session supervision. Verify backend lock release and preserve interrupted migration errors. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Send public driver cancellation before aborting the transport, within one absolute cleanup deadline. Separate borrowed migration execution so its operator keeps sole session supervision. Verify backend lock release and preserve interrupted migration errors. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Supervise bounded Postgres phases independently, reserving cleanup inside each existing deadline. Send public close while query and transaction work are alive, then cancel and reap within that same bound. Borrowed checks and trigger DDL leave failed-session cleanup to their owner. Bound the lock owner's close and preserve its original failure across cleanup errors or repeated cancellation. Keep wait floors and normal busy retries. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Supervise bounded Postgres phases independently, reserving cleanup inside each existing deadline. Send public close while query and transaction work are alive, then cancel and reap within that same bound. Borrowed checks and trigger DDL leave failed-session cleanup to their owner. Bound the lock owner's close and preserve its original failure across cleanup errors or repeated cancellation. Keep wait floors and normal busy retries. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Supervise attach, prepare, removal and backout on their exact lock backend. Keep cleanup inside phase deadlines, forward fatal guard exceptions after close, and retry mode contention only after acknowledged rollback and ownership checks. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep the existing store checks and failure envelope while giving each refused observation its own source site. Protect the native test child with the same kill and wait ownership as neighboring tests. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
The initialization server can answer native queries before its final HTTP listener starts. Probe the published endpoint through the existing bounded creator command. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
A zombie leader can still report multiple threads. Keep polling within the existing allowance until the complete asserted exit state is observed. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
The fixture owns the child through timeout and cancellation. Keep its existing wait budget and always collect that exact process before store cleanup. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Cancel and join the owned task before dedicated sessions close on an interrupted control operation. Preserve the contention checks and wait allowances. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Give each in-process test its own real gate while keeping shutdown permanent within the scenario. A private patch context preserves the fixture through explicit test monkeypatch cleanup. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
A canceled head-lock query leaves its cancellation mechanism and query plan unknown. Record closed reasons and a bounded fresh planning observation while retaining the original failure and cleanup contract. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Keep the production query and statement budget while reproducing the shared-identity workload in an isolated temporary table. Exact-row, ordering and identity controls distinguish a timeout from an incorrect match. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Enroll before accepting org work, retain the permanent lifecycle through record cleanup, and preserve the first failure through both store closes. Classify the external feed table builder by its exact SQL sink. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
A large absent batch can merge on a partial identity and exhaust the existing statement budget. Sort complete identities before correlated row locks so the batch cannot flatten into that join. Retain all six tenant predicates, the transaction and lock order, and the unchanged native regression and deadlines. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Register each owned database and session as it is acquired. Attempt every cleanup when setup or teardown fails and preserve the original setup or body error. Keep all native assertions and deadlines unchanged. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
A scan can outlive its ATTACH backend and recreate tenancy through the pool after fenced cleanup. Use the existing owner transaction for each unchanged batch and stop after an unknown commit. Keep synthetic request state scoped in its test, account for the tracker owner, and attempt every native fixture cleanup. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
A stalled backend can outlive bounded client cleanup. Require exact-holder verification when release cannot be proved. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Backend termination can remain unproved after cleanup, and later-cell removal verifies paired stores. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Retain bounded administration, native regression baselines and generated surfaces while adopting the current attachment and lifecycle corrections. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Use a complete principal and healthy control, and give blacklist regressions valid fixture org labels so native execution reaches the intended assertions. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
6 of 7 tasks
Retain connected-account cleanup before subject-revocation completion and preserve bounded cell actions and native successor regressions. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Preserve the authorized row fields and refuse an unrepresentable successor before writing. Correct the composed type and snapshot contracts, keep budget validation details private, and report allowlisted metadata for native fixture bootstrap failures. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Remove unused forwarding functions from the intent parser. Keep command execution and durable state in the modules used by the CLI. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Remove two unused context-key sets. Request construction and exact context validation continue to share the active context helpers. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Use the active saved-command path for resume and expose only supported request models. Remove two definitions with no consumers. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Cover capacity and backend preconditions, storage failures and the second admission check before local adoption. Remove an unreachable duplicate backend condition while retaining both checks around storage. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
The empty prefix fails settings validation before migration starts. Check both private cell environments against the real settings model before native execution. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
The shared HTTP exception handler adds auth classification and non-retryability to coded 403 responses. Keep the native refusal check exact and retain its unchanged-row assertion. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Retain the provisioning invalidation boundary around ordinary and bounded descendant revocation writes. Keep the complete command surface and regenerate its CLI snapshots from the merged parser. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
Await bounded public close while the failed initializer still owns its raw session. Preserve the original failure and report unknown closure when cancellation or cleanup interrupts disconnect. Signed-off-by: zlareb1 <lareb.zafar@clickhouse.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related: #735
Changelog category (leave one):
Changelog entry (one or two sentences a user of Loom can read without the diff; the release pull request collects these into the release notes):
loom cell eachruns supported administration commands across registered cells and records each result in a private resumable outcome file.loom cell automationverifies the automation and recovery keys before registering their public identities.Why
Administration that applies to a deployment must reach every cell without granting the router access to cell private keys. The coordinator uses the directory's current identities and each cell's existing authentication and administration owners. An uncertain mutation stops automatic continuation rather than being replayed.
Scope and risk
SSL_CERT_FILE, with proxies and redirects disabled. The command's explicitca_fileremains separate from server trust.The main risks are incorrect bot grants, a directory change during a run, partial business effects and an unknown response after dispatch. The command checks the current target before each attempt and after the run. Operators must inspect ambiguous effects through the recovery procedure; the outcome file is not an idempotency guarantee.
Compatibility, rollout and rollback
docs/operations/cell-administration.mdand the generated HTTP reference. The Python administration route table is generated with them. Existing data-plane operation fields are unchanged.LOOM_MAX_REQUEST_BYTES=0disables the global body cap; fixed cell administration caps still apply. No new setting or database migration is added.loom cell automation. Supply a private credential manifest and outcome path toloom cell each.Tests
pytest -q -n 0 tests/unit/test_error_envelope.py::test_a_coded_http_exception_carries_its_class_and_a_plain_one_is_untouched tests/unit/test_error_envelope.py::test_a_typed_http_error_carries_its_class tests/unit/test_cell_admin_subject_route.py::test_subject_route_refuses_authoff_before_business. These cover the shared envelope and actual authority guard, not the complete native case.AI assistance in the code (leave one; writing this description with AI is fine):
AI-generated (an agent wrote most of the diff): Claude Code and Codex. Independent reviews check the authentication, bounded operation, resume and tenant-isolation contracts.
I can explain every line of this diff and how it interacts with the rest of Loom.
Checklist