diff --git a/.gitignore b/.gitignore index 923ee9d..025421e 100644 --- a/.gitignore +++ b/.gitignore @@ -66,6 +66,4 @@ tmp/ temp/ examples/ -# Documentation examples are reviewed source, not local sample projects. -!docs/examples/ lambda-exporter/ diff --git a/CHANGELOG.md b/CHANGELOG.md index f516f32..5d55207 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,13 +4,45 @@ Changes are recorded in [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) ## [Unreleased] -## [0.9.0] +## [0.7.0] -0.9 brings everything planned for 1.0. It stays below 1.0 until the stability promises (frozen JSON output, the `rebuild_table` guarantee) have run on real workloads. A normal `pip install -U` reaches it from 0.5.x, so read this before upgrading: +0.7 waits for the work a migration starts. `up` doesn't record a migration as applied until the mutations and `ON CLUSTER` DDL its statements started have finished, and it shows their progress while it waits. A normal `pip install -U` reaches it from 0.6, so read this before upgrading: + +1. **`up` now blocks until that work finishes, with no default timeout.** If a CI job's time limit stops it mid-wait, ClickHouse keeps working, and the next `up` waits for that work before it runs the migration again. `--timeout SECONDS` sets a limit of your own. +2. **A rerun runs the stopped migration again from its first statement, but only on its own when every statement passes the idempotency gate without a waiver.** An interrupted migration with a waived statement, which includes every `UPDATE`, `DELETE` and `INSERT ... SELECT`, stops the next run until you check what the earlier one changed and name it with `--rerun-interrupted REV`. CI jobs that retry `up` after a timeout need that step. +3. **The gate now reads more of your migrations.** SQL in `text("...")` and SQL sent through `op.get_bind().execute(...)` or `exec_driver_sql(...)` is checked, and SQL built at run time needs a waiver. `lint` and `up` can report migrations that passed before; waive them or keep them behind `lint.gate_baseline`. +4. **Two grants, only where `bootstrap` couldn't grant `SELECT ON system.*`**, such as on ClickHouse Cloud. As the admin user, replacing `` with your project name: + ```sql + GRANT CURRENT GRANTS(SELECT ON system.parts) TO _migration_role; + GRANT CURRENT GRANTS(SELECT ON system.replicas) TO _migration_role; + ``` + `system.parts` gives the progress bar its bytes and time estimate; without it the bar counts parts. `system.replicas` is read only for self-hosted replicated tables, so Cloud needs just the first. +5. **No `upgrade-env` this time.** The environment 0.6 wrote is current. `up` creates one small table, `_ch_migrate_in_progress`, in the project database. + +To stay on 0.6, pin `ch-migrate-cli==0.6.0`. + +### Changed + +- `up` waits before it records a migration as applied. After each statement that can write, it polls `system.mutations` on the migration session until the project database has no unfinished mutation on any replica that has to apply it, instead of holding a request open with `mutations_sync`. Mutations another writer started count too, and ones already queued are waited for before the first migration runs. The version table's own mutations stay with its 0.6 bookkeeping. +- `ON CLUSTER` statements wait inside ClickHouse: `up` sends them with `distributed_ddl_output_mode = 'throw'` and a limit of 270 seconds, or what's left of `--timeout` if that's less, so a host that fails or doesn't finish in time fails the migration with ClickHouse's message. +- A failed mutation fails the migration with ClickHouse's `latest_fail_reason` and prints the scoped `KILL MUTATION` statement for an operator; `ch-migrate` never kills a mutation itself. A failed mutation from before the run blocks it the same way. After a kill, fix the SQL and run `up` again, naming the revision with `--rerun-interrupted` if it's waived. +- `up` runs Alembic in its own process, so killing `up` can't leave a child process advancing the version table. `down` and a direct `alembic upgrade` go through the same environment and wait too; `--timeout` exists only on `ch-migrate up`. +- The idempotency gate checks literal SQL in `text("...")` and in `execute`/`exec_driver_sql` on any connection, requires a waiver for SQL that only exists at run time, and runs from `env.py` for a direct `alembic upgrade`. + +### Added + +- `up --timeout SECONDS`: one waiting budget for the whole run. When it runs out, `up` exits 1, names what is still pending and leaves the work running; the next `up` waits on it again, then runs the migration. +- Progress while `up` waits: in a terminal, a live bar per mutation weighted by bytes from `system.parts`, with parts and time left, and one row per replica. Piped output gets plain lines when a wait starts, at each 10% or every 30 seconds, and when it finishes. Without access to `system.parts` the bar counts parts only. +- `up --rerun-interrupted REV` and `down --rerun-interrupted REV` (`-x rerun_interrupted=REV` with `alembic`): run an interrupted migration again after checking what its earlier run changed. Each migration is noted in `_ch_migrate_in_progress` while it runs, created with the version table's engine. +- `bootstrap` grants `SELECT` on `system.replicas` and `system.parts`. + +## [0.6.0] + +0.6 runs migrations on ClickHouse's official clickhouse-connect dialect in one session-safe connection, and stops a migration from running twice. A normal `pip install -U` reaches it from 0.5.x, so read this before upgrading: 1. **Python 3.10 or later.** On Python 3.9, pip keeps installing 0.5.x. -2. **Run `ch-migrate upgrade-env` in each project.** `up`, `down`, `status` and `history` refuse the old environment until you do. -3. **Check `ch-migrate lint ENV`.** The idempotency gate can now block `up` for revisions written after the `upgrade-env` baseline. +2. **Run `ch-migrate upgrade-env` in each project.** `up`, `down` and `history` refuse the old environment until you do. It replaces the whole `migrations/env.py` and saves the old file as `migrations/env.py.bak`; reapply any customizations from the backup. +3. **Check `ch-migrate lint ENV`.** The idempotency gate can now block `up`, but only for revisions written after the baseline that `upgrade-env` records. To stay on 0.5.x until then, pin `ch-migrate-cli==0.5.1`. @@ -18,38 +50,15 @@ To stay on 0.5.x until then, pin `ch-migrate-cli==0.5.1`. - **Python 3.10 or later is required.** 0.5.x is the last line that supports Python 3.9. - Migrations run on the official clickhouse-connect dialect (`clickhousedb://`) in one checked HTTP session per run, so a standalone `SET` applies to later statements. Run `ch-migrate upgrade-env` after upgrading; old environments are refused before connecting. +- The migration session uses its own single-connection pool. ClickHouse Cloud routes each TCP connection, not each session id, to a replica, so the session stays on the connection that created it. A request that reaches a replica without the session fails with `SESSION_NOT_FOUND` instead of running with default settings. - The idempotency gate can block `up` on projects upgrading from 0.5. `upgrade-env` sets the baseline, so only revisions written after it are gated. -- `up` now blocks until the mutations and `ON CLUSTER` DDL it starts have finished, and `--timeout` bounds that wait. - The version table no longer lets an interrupted run re-run a migration. 0.5.x and earlier can, if `up` runs again before a delayed version update applies. - `lint ENV` analyzes only pending revisions after the gate baseline; static `lint` checks every revision after it. - `up` refuses idempotency gate errors before applying migrations; other lint findings are nonblocking warnings. - `upgrade-env` records current script heads as a comment-preserving `lint.gate_baseline`. -- Retired the incorrect `large_table_mutation` rule and row threshold. Legacy configuration is ignored with a deprecation warning pointing to `plan`. -- Rebuild preflight reads active queued writers' per-query settings when query-log evidence has not flushed. Host/query identity and unknown-writer refusals remain enforced. -- Rebuild preflight filters and groups the writer log in the server, by host, user and async settings. It no longer refuses on a busy service whose last hour of inserts into other tables exceeds the row bound. ### Added -- Continuous-writer rebuild acceptance for synchronous, interrupted, replicated, and both async acknowledgement modes, with server-time duplicate attribution, real error-741 retries, and exact reported-loss accounting. - -- SQL-first `new --table T --rebuild`, using portable live DDL or a commented offline placeholder and an irreversible guarded revision. Rebuild plans provide shell-quoted suggestions in human output and optional JSON. -- `--exchange` remains available but warns that its copy cannot preserve concurrent writes. - -- Rebuild dependent checks against the replacement schema, safe cleanup on pre-copy validation rejection, and post-swap dictionary reloads. Source/TO materialized views and inner-engine views keep their existing definitions. -- Guarded online `rebuild_table`: same-database snapshot/staging helpers, durable partition checkpoints, all-replica swap barriers, a retained rollback table, and UUID-attributed async rejection reporting. -- `rebuild_table` on ClickHouse Cloud. Helpers prove ownership with an owner-token `COMMENT` and a read-back UUID, because Cloud rejects chosen UUIDs. The swap relies on `EXCHANGE` draining the old async queue, since Cloud forbids the flush, so the async rejection report is a lower bound (`"complete": false`, with the replicas read). The migration must declare `writers_on_this_service_only=True`; until it does, preflight refuses with `cloud_writers_undeclared`. `grace_seconds` (default 10) pauses after the copying view reaches every replica. -- The migration session uses its own single-connection pool. ClickHouse Cloud routes each TCP connection, not each session id, to a replica, and sharing the driver's default pool let other in-process clients open a second connection under the session. Its next request could then reach a replica without the session and fail with `SESSION_NOT_FOUND`. -- Rebuild snapshot and stage helpers are plain `MergeTree` without TTL, in the source's replication family, instead of copies held still by `SYSTEM STOP MERGES`. A server restart silently undoes `STOP MERGES`, after which a Replacing, Collapsing, Summing or Aggregating snapshot could merge rows away and leave the rebuild unable to finish. Plain merges keep every row. -- After the swap, `rebuild_table` checks `system.part_log` for rows that reached only the old table. An insert that resolves the table just before `EXCHANGE` and builds its view chain just after it skips the copying view. Synthetic trials hit this with async inserts, `wait_for_async_insert=1` included, in about 2% of swaps on 26.3 and Cloud. Any such rows stop the run with their count and query IDs and keep the lock until the operator copies them and releases it. The report records them as `"stranded"`, and preflight refuses hosts without `system.part_log`. Bootstrap grants `SELECT` on it. -- Non-takeover rebuild locks with independent heartbeats. Interrupted owners require explicit operator reconciliation and release before checkpoint resume; heartbeat expiry never authorizes a competing runner. -- Read-only `plan ENV [--json]`: ordered pending statements, classifications, exact available column bytes versus whole-part ceilings, dependencies, lint gate status, shared rebuild preflight, and a versioned JSON schema. -- Rebuild preflight checks replicas, physical-transfer definitions, writer profiles and rotated logs; reports per-host capacity and mutation hazards; and requires explicit acknowledgement of unacknowledged async-write loss. -- Default mutation waiting with durable ownership, per-replica completion, same-session progress, an optional invocation timeout, and fail-closed unknown-outcome recovery. Upgrades now run in the owning CLI process. -- Resumable insert-first version bookkeeping and fresh waiting-journal generations after downgrade, without replaying completed revision writes. Every version-row INSERT, including those `down` writes, carries its own deduplication token, so Replicated and Shared MergeTree cannot drop a re-inserted earlier head and leave the version table empty. -- Owned `ON CLUSTER` queue waiting, per-host failure/timeout diagnostics, and acknowledgement-loss recovery without re-enqueueing DDL. Mutation-producing DDL must finish both its queue and mutation barriers. -- Versioned `--json` output and published JSON Schemas for status, history, lint, and diff, with machine-readable errors, graph-resolved applied revisions, explicit waiver reasons, and CI-friendly exit codes. -- Server-backed statement classification with live type comparisons and a 50-statement corpus. Every corpus ALTER is checked against real mutation records or the server's in-place-change rejection. -- A SHA-pinned GitHub Actions drift gate that preserves the JSON report on failure; its install, credential, comparison, and exit steps run against the integration fixture for both matching and out-of-band schemas. - Reviewed idempotency corpus covering DDL, mutations, inserts, and partition operations. - Required-reason `ch-migrate: allow-non-idempotent` statement waivers, reported as INFO. - Gate findings say how to make the flagged statement safe to run twice, using its own names: the exact `IF NOT EXISTS` or `IF EXISTS` rewrite, or what a retry would do and the usual fix, such as `TRUNCATE` before a backfill or `REPLACE PARTITION` instead of `ATTACH PARTITION ... FROM`. @@ -57,6 +66,7 @@ To stay on 0.5.x until then, pin `ch-migrate-cli==0.5.1`. - An isolated one-shard/two-replica integration harness with embedded Keeper, owned Docker network cleanup, node stop/start, and distributed-DDL timeout/catch-up coverage. - Deployment-aware version-table engines: replicated state for self-hosted replicated databases and configured clusters, preserving existing tables without automatic conversion. - Replica read barriers and failed version-mutation checks, plus live status warnings and an offline upgrade advisory for manual conversion. +- Every version-row INSERT, from `up`, `down` and `stamp`, carries its own deduplication token, so Replicated and Shared MergeTree cannot drop a re-inserted earlier head and leave the version table empty. ### Fixed @@ -71,7 +81,7 @@ To stay on 0.5.x until then, pin `ch-migrate-cli==0.5.1`. ## [0.5.0] -0.5.x is the last release line that supports Python 3.9. 0.9 requires Python 3.10 or later, because it moves to the official clickhouse-connect dialect. +0.5.x is the last release line that supports Python 3.9. 0.6 requires Python 3.10 or later, because it moves to the official clickhouse-connect dialect. **Renamed.** The PyPI package is now `ch-migrate-cli` (install with `uv tool install ch-migrate-cli`) and the import package is `ch_migrate`. Releases up to 0.4.1 were `clickhouse-alembic` / `clickhouse_alembic`; Alembic is how `ch-migrate` stores revisions, not what it is. The command is still `ch-migrate`. Uninstall `clickhouse-alembic` before installing `ch-migrate-cli`, because both install the `ch-migrate` command. Existing projects keep working: `clickhouse_alembic` remains as an alias that warns where it's imported and is removed in 1.0. To move over, replace `clickhouse_alembic` with `ch_migrate` in your migration files and run `ch-migrate upgrade-env`. @@ -106,4 +116,4 @@ To stay on 0.5.x until then, pin `ch-migrate-cli==0.5.1`. ### Known issues -- If the version-table update is delayed and `up` is run again before it applies, a migration can run twice. Fixed in 0.9; see Q2 in [the dialect spike](docs/design/2026-10-02-dialect-spike.md#q2-the-version-table). +- If the version-table update is delayed and `up` is run again before it applies, a migration can run twice. Fixed in 0.6; see Q2 in [the dialect spike](docs/design/2026-10-02-dialect-spike.md#q2-the-version-table). diff --git a/README.md b/README.md index 4b6c765..3c9f2f7 100644 --- a/README.md +++ b/README.md @@ -23,8 +23,15 @@ This README describes the source checkout, which may be ahead of PyPI. To try an ## Upgrade from 0.x -0.9 requires Python 3.10+; 0.5.x is the last line supporting Python 3.9. -A loose requirement such as `ch-migrate-cli` or `~=0.5` upgrades to 0.9; pin +**From 0.6:** 0.7 needs no `upgrade-env`. `up` now waits for the mutations and +`ON CLUSTER` DDL its statements start (see [Migrations that wait](#migrations-that-wait)), +so give CI jobs time to finish or pass `--timeout`. Where `bootstrap` couldn't grant +`SELECT ON system.*`, such as on ClickHouse Cloud, add the grants listed in the +[CHANGELOG](CHANGELOG.md). A loose requirement such as `ch-migrate-cli` upgrades to +0.7; pin `ch-migrate-cli==0.6.0` to stay on 0.6. + +**From 0.5:** 0.6 and later require Python 3.10+; 0.5.x is the last line supporting Python 3.9. +A loose requirement such as `ch-migrate-cli` or `~=0.5` upgrades past 0.5; pin `ch-migrate-cli==0.5.1` until each project has run the command below. After upgrading the package, run this in each migration project: @@ -36,8 +43,9 @@ ch-migrate upgrade-env package. The old file is saved as `migrations/env.py.bak`; repeating the command on the current shim leaves that backup intact. Review any old customizations. Existing Python migrations and version tables continue to work without edits. -`up`, `down`, `status`, and `history` refuse old environments before connecting -and tell you to run `upgrade-env`. +`up`, `down`, and `history` refuse old environments before connecting and tell +you to run `upgrade-env`; `status` still reports, so CI jobs that use it as a +non-blocking reporter keep passing. One HTTP session spans a migration run, including later revisions in that run. A standalone `SET` therefore carries into following revisions. Prefer a @@ -46,12 +54,6 @@ statement's `SETTINGS` clause when the change should apply only to that query. seconds (default `1800`). If that session expires, the next request fails rather than silently recreating a session with default settings. -On ClickHouse Cloud a session lives on one replica, and Cloud routes each TCP connection, not -each session, to a replica. ch-migrate keeps a run on one dedicated connection. If that -connection drops, for example after a migration pauses client-side for more than Cloud's -10-second keep-alive, the run stops with `SESSION_NOT_FOUND` instead of continuing without -your settings. - ## Quick start Use a dedicated ClickHouse test server. The example uses local HTTP; replace the host and port with your server's address. For HTTPS, set `secure: true` and the HTTPS port (usually `8443`). Never run a trial migration against a shared or production database. @@ -178,14 +180,13 @@ New `alembic_version` tables use `ORDER BY version_num`. The runtime inspects | Atomic database with `cluster` configured | `ReplicatedMergeTree` created `ON CLUSTER`, with a Keeper path containing the database name and shard macro | | Single server | `MergeTree` | -Version advancement remains **insert the new row, then delete the old row**; -there is no asynchronous version UPDATE. Online `up` journals that bookkeeping -and polls its owned DELETE rather than holding an HTTP request open with -`mutations_sync`. If interrupted, both rows can remain. A subsequent `up` -finishes the proven bookkeeping without running the revision again. Overlapping -heads without an owned journal receipt are still refused. Failed version -mutations report their reason and an operator-only KILL statement; ch-migrate -never kills them itself. +The official Alembic implementation advances a version by **inserting the new +row, then deleting the old row with `mutations_sync = 2`**. It does not issue an +asynchronous version UPDATE. If the process stops while the delete is pending, +both rows remain and Alembic refuses their overlapping history rather than +repeating the migration. Let the version mutation finish, then retry. A +server-reported failed version mutation stops the run with its reason; ch-migrate +never kills that mutation. Before reading replicated heads, the tool catches up replicated database metadata and inserted version parts. The table barrier uses @@ -235,97 +236,94 @@ is separate from the local suite; the local replicated harness has one shard. ### Migrations that wait -`up` waits for the mutations its statements create before completing a revision. -It uses the same checked ClickHouse session for submission and polling, so prior -`SET` statements stay effective. `op.execute`, raw `op.get_bind()` execution, and -`run_sql` all pass through the cursor boundary. Upgrades run in the CLI process: -killing that process cannot leave a child Alembic runner advancing revisions. +`up` doesn't record a migration as applied until the work its statements started +has finished. After each statement that can write, it polls `system.mutations` on +the same checked ClickHouse session, so earlier `SET` statements stay in effect. +`op.execute`, raw `op.get_bind()` execution and `run_sql` all pass through that +check. Upgrades run in the CLI process, so killing it can't leave a child Alembic +runner advancing revisions. `down` waits the same way. ```bash ch-migrate up dev # No default waiting timeout ch-migrate up dev --timeout 120 # One waiting budget across the invocation ``` -Progress names the table, mutation ID, host, remaining parts and elapsed time. -A terminal uses a live line; CI gets plain state changes and periodic summaries -on stderr. Zero remaining parts alone does not count as completion: every -required replica must report its owned mutation finished. Configure `cluster` -for self-hosted replicated tables, or use a Replicated database's automatic -cluster. Missing replicas or missing evidence are never treated as success. - -An active foreign mutation ahead of ours is reported and waited before more work -is submitted. A failed predecessor stops the run. Foreign work queued after our -completion barrier is not claimed as ours and does not hold our revision open. - -Ownership is persisted in `_ch_migrate_journal` **before** SQL submission. The -journal follows the version table's deployment policy and uses the same session. -It records revision/statement identities, table UUIDs and progress receipts. -Run only one migration runner per database, using a CI concurrency key or an -external lock: this journal is not a distributed lock. Do not delete or restore -it independently of the database state it describes. - -After a timeout, dropped connection or killed process, ClickHouse may still be -working. Run the same `up` again: it skips proven completed statements and -reattaches to its known unfinished mutation instead of issuing it again. -Successful downgrade starts a fresh journal generation so a later upgrade -actually executes. A changed unresolved statement or replaced table is refused. -An explicitly rejected repeat-safe metadata statement can be repaired and retried; -transport errors do not establish rejection. - -**Unknown outcome:** if ownership/completion evidence is missing or expired, -`up` exits nonzero, does not reissue the statement and does not complete the -migration. The error names the journal key, token and recorded UUIDs and provides -read-only inspection queries. Operator recovery is deliberately not a retry flag: - -1. Stop migration runners and quiesce other writers as needed. Back up the - affected journal key and relevant data before making changes. -2. Establish whether the original statement took effect using independent data, - schema and server evidence on every required replica. An absent mutation row - is not proof either way. -3. Reconcile that **specific** statement and journal key in a reviewed manual - operation. For a proven applied operation, its completed receipt must carry - the actual post-operation UUID map (`after_target.uuids`, or null for a dropped - target). For a proven unapplied operation, remove only that key's attempt - records after restoring any partial effects, and wait for that journal - change on every replica. Preserve the other completed statement receipts. -4. Recheck the recorded heads and rerun `up`. Never blindly stamp the revision, - delete the whole journal, or replay a non-idempotent statement to guess its - outcome. - -**Failed mutation:** `latest_fail_reason` stops the run and prints a scoped -`KILL MUTATION` statement for the operator. The tool does not execute it. Fix the -cause and reattach, or review a cancellation and reconcile its effects; killing -a mutation is not proof that it completed. - -`MODIFY TTL` has a separate metadata/materialization boundary: when materialization -is enabled, the runtime journals the metadata change with automatic materialization -disabled, then submits a separately owned `MATERIALIZE TTL`. This avoids confusing -TTL's comma-separated rules with ALTER actions. Session and per-query -`materialize_ttl_after_modify = 0` are honored. Parameter batches are executed -sequentially through the same single-statement path, with a receipt and completion -barrier for each parameter set; intermediate TTL effects are not collapsed. -Lightweight DELETE keeps its row-mask semantics; lightweight UPDATE is synchronous -patch work, not a fabricated `system.mutations` record. - -For `ON CLUSTER` statements, the same journal first records a unique distributed -DDL marker. `up` submits without a server-side synchronous DDL wait, then polls -`system.distributed_ddl_queue`. Every configured target host must report -`Finished` **and** exception code zero. A failed host is named with its error; -a down host remains pending until it returns or the shared `--timeout` expires. -The tool neither deletes the queue entry nor resubmits the DDL on rerun. - -The queue entry is found by its recorded `log_comment`, not by matching SQL text -(the server can insert UUIDs into that text). Caller log comments are preserved -in the journal. A lost acknowledgement can reattach to the existing entry; -missing/expired queue evidence follows the unknown-outcome procedure above. -Mutation-producing distributed ALTERs must satisfy **both** barriers: DDL execution -on all hosts and completion of their owned mutations. TTL metadata and explicit -materialization retain separate queue receipts. - -Single-node statements without `ON CLUSTER` do not consult that queue. Ordinary -Cloud DDL does not need `ON CLUSTER`; Cloud qualification remains separate from -the local harness. Re-run bootstrap when upgrading a restricted migration user -to obtain the available cluster and distributed-queue inspection grants. +What `up` waits for: + +- **Every unfinished mutation in the project database,** including one another + writer started. Before the first migration runs, `up` waits for any already + queued, and a failed one blocks the run. The version table's own mutations are + left to [its bookkeeping](#the-version-table). +- **Each replica that has to apply it.** Zero parts left isn't completion on its + own: the mutation must report `is_done`. A ReplicatedMergeTree mutation must be + finished on every replica, and a replica that hasn't received it yet is still + pending. Configure `cluster` for self-hosted replicated tables, or use a + Replicated database's automatic cluster; without either, `up` refuses a table + with more than one replica. SharedMergeTree on ClickHouse Cloud reports one + shared mutation. +- **`ON CLUSTER` DDL on every host.** `up` sends these statements with + `distributed_ddl_output_mode = 'throw'`, so ClickHouse answers once every host + has run the statement, and a host that fails or doesn't finish in time fails the + migration with ClickHouse's message. The limit is 270 seconds, below the + client's read timeout, or what's left of `--timeout` if that's less. Mutations + the statement starts are then waited for as above. + +Statements ClickHouse already holds until they finish, such as a type-changing +`MODIFY COLUMN`, a `MODIFY TTL` that materializes, or a lightweight `DELETE`, +still hold their request as they did in 0.6. + +While it waits, `up` shows progress on stderr. In a terminal, each mutation gets a +live bar weighted by bytes, with its parts and an estimate of the time left: + +```text +→ Applying 3f9c2a1b backfill_digest + ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━ 18.2 / 41.0 GiB · 23/41 parts · 1:12 · ~1:30 left +✓ 3f9c2a1b backfill_digest mutation_7.txt · 41 parts · 41.0 GiB · 2:03 +``` + +Self-hosted replicated tables get one row per replica. When output isn't a +terminal (CI logs, pipes, coding agents), `up` prints plain lines instead: one +when a wait starts, one at each 10% of the parts or every 30 seconds, and one +when it finishes: + +```text +→ Applying 3f9c2a1b backfill_digest + waiting on mutation_7.txt (demo_dev.events): 0/41 parts + waiting on mutation_7.txt (demo_dev.events): 5/41 parts · 0:14 +✓ 3f9c2a1b backfill_digest mutation_7.txt · 41 parts · 2:03 +``` + +Part sizes come from `system.parts`. If the migration role can't read it, the bar +counts parts only and says so once; progress never changes whether `up` succeeds. + +**When `up` stops short.** After a timeout, Ctrl-C, dropped connection, killed +process or failed statement, ClickHouse keeps working and the migration isn't +recorded. `up` and `down` note each migration in `_ch_migrate_in_progress` when it +starts and clear the note once its version is recorded, so the next run knows the +migration was interrupted. That run waits for unfinished work, then: + +- **Runs the migration again on its own** when every statement in it passes the + idempotency gate without a waiver, so a repeat leaves the same result. +- **Stops before sending anything** when the migration has a waived statement or + SQL the gate can't read, because that statement may already have applied. It + names each one. Check what the earlier run changed, fix the SQL if needed, then + name the revision: `ch-migrate up ENV --rerun-interrupted REV` (or `down`), or + `-x rerun_interrupted=REV` with `alembic`. Naming it accepts the repeat. + +Every `UPDATE`, `DELETE` and `INSERT ... SELECT` needs a waiver, so an interrupted +backfill always takes that second path. + +**Failed mutation.** ClickHouse's `latest_fail_reason` stops the run, and `up` +prints the scoped `KILL MUTATION` statement for an operator; ch-migrate never +runs it. ClickHouse keeps retrying a failed mutation, and later mutations on that +table queue behind it. If the cause is temporary, such as memory or disk, fix it +and run `up` again. If the statement itself is wrong, kill the mutation, fix the +SQL and run `up` again, naming the revision with `--rerun-interrupted` if it's +waived. Killing a mutation doesn't undo the parts it already rewrote. + +Run one migration runner per database, using a CI concurrency key or an external +lock; the in-progress notes aren't a lock. Re-run `bootstrap` when upgrading a +restricted migration user to obtain the `system.replicas` and `system.parts` grants. ### Re-runnable migrations @@ -333,7 +331,14 @@ to obtain the available cluster and distributed-queue inspection grants. them. Idempotency and standalone-SET errors refuse the whole run and print the source file, line, statement, and suggested fix. There is no command-line bypass; `--skip-mv-check` does not skip this gate. Other findings are warnings in `up`; standalone `lint` -retains their configured severities. +retains their configured severities. A direct `alembic upgrade` runs the same gate +from `env.py` before its first migration. + +The gate reads SQL in `run_sql` files and in literal strings, f-strings and +`text("...")` passed to `op.execute`, or to `execute` and `exec_driver_sql` on a +connection such as `op.get_bind()`. SQL that only exists at run time, such as a +variable, a built string or `run_sql(name)`, can't be checked, so it needs a +reasoned waiver like any other statement the gate can't vouch for. Static lint rejects a standalone `SET` when `migrations/env.py` lacks the `CH_MIGRATE_ENV_VERSION = 2` marker: that statement would be ignored on the old @@ -367,7 +372,7 @@ INSERT INTO {db}.target SELECT id FROM {db}.source; ``` For inline Python SQL, use the same directive in a `#` comment directly above -`op.execute`. Empty reasons remain errors. Waivers print as INFO with their +the call that sends it. Empty reasons remain errors. Waivers print as INFO with their reasons in `lint` and `up`; they do not make the statement idempotent. `MODIFY`, `MATERIALIZE`, `CLEAR COLUMN`, `TRUNCATE`, `OPTIMIZE`, `SYSTEM`, grants, revokes, `REPLACE PARTITION`, attaching or detaching a whole partition, and @@ -376,11 +381,12 @@ The reviewable classification corpus is `tests/corpus/idempotency.yaml`. **Recovery:** fix the cause of a partial failure, then run `ch-migrate up ENV` again. Earlier idempotent statements can run again; the version advances only -when the revision completes. On ClickHouse Cloud, a column that the failed attempt -dropped or renamed stays locked until that change's mutation finishes: a retry -before then fails with "affected by mutation ... which is not finished yet", even -with `IF EXISTS`. Wait for it in `system.mutations`, then run `up` again. Before retrying waived operations, review what the -earlier attempt actually changed. +when the revision completes. A dropped or renamed column stays locked until that +change's mutation finishes, and `up` waits for it before running the next +statement, so a retry after a later failure doesn't hit "affected by mutation ... +which is not finished yet". If `up` was stopped while it waited, run it again: it +waits for that mutation, then runs the revision again, unless the revision has a +statement the gate can't vouch for (see below). **Existing history:** `upgrade-env` records the current local script head(s) in `config.yaml` as `lint.gate_baseline`, preserving YAML comments. It records @@ -441,7 +447,7 @@ def downgrade(): ### Exchange and dictionary patterns -`new --exchange --table NAME` is deprecated. It still generates the legacy shadow-table, copy, exchange, and drop scaffold, and prints a warning pointing to `--rebuild`. Rows written during the legacy copy can be lost: coordinate or pause writers if you deliberately retain that workflow. For online changes, use the guarded `--rebuild` workflow below. The legacy scaffold remains irreversible because it drops the old table. +`new --exchange --table NAME` generates the existing shadow-table, copy, exchange, and drop scaffold. Coordinate or pause writers: this copy-and-swap pattern alone does not preserve inserts arriving during the copy. It is not an online-rebuild guarantee. Review the generated SQL and column mapping before applying it. The scaffold is marked irreversible because it drops the old table. The generated copy and exchange statements require explicit, reasoned waivers before `up` accepts them. The generator does not waive them automatically. @@ -467,353 +473,12 @@ def upgrade(): create_dictionary("history/dictionaries/dict_users/001_create.sql") ``` -## Changing a sorting key or engine - -`op.rebuild_table()` builds a replacement while writers keep inserting, then -swaps it into the original name. Its guarantee is **no acknowledged row is lost, -and a small number may be duplicated**. Duplicates can come from rows committed -or flushed between copying-view creation and snapshot completion, in server time. -ReplacingMergeTree variants may collapse those copies on merge; other engines -can retain them. - -**The exception is unacknowledged async inserts:** rows queued with -`async_insert = 1, wait_for_async_insert = 0` at the swap may be rejected by -ClickHouse. Such writers require `allow_unacknowledged_async_loss=True` in the -migration file. There is no command-line bypass. Writers that wait for -acknowledgement should retry error **741** (`TABLE_UUID_MISMATCH`) and other -transient rebuild errors. Writers must not ignore materialized-view errors. - -**One race ClickHouse leaves open is detected, not prevented.** An insert that -resolves the table just before `EXCHANGE` and builds its materialized-view chain -just after it writes only to the old storage, which becomes the rollback table; -the copying view never sees it. In synthetic trials with eight busy writers, it -hit async inserts, including `wait_for_async_insert = 1`, in about 2% of swaps. -Synchronous inserts were never hit, but that does not prove they are immune. -After the swap, `rebuild_table` -reads `system.part_log` for parts created in the old storage by queries that -wrote nothing to the new table. If it finds any, the run stops with the row -count and query IDs and keeps its lock; see -[Recover rows stranded at the swap](#recover-rows-stranded-at-the-swap). -Preflight refuses when a host has no `system.part_log`. - -**On ClickHouse Cloud** the rebuild runs too, with three differences: - -- **Declare your writers.** Cloud's compute-compute separation hides other services' inserts - from this one, and an insert of theirs could be missing from the rebuilt table. Once every - writer to the table uses the service you migrate through, set - `writers_on_this_service_only=True` in the `rebuild_table` call; `new --rebuild` writes it - as `False` for you to change. Until then `plan` reports, and `up` refuses with, - `cloud_writers_undeclared`. After the copying view reaches every replica, the rebuild - waits `grace_seconds` (default 10) before draining inserts. -- **No forced queue flush.** Cloud forbids `SYSTEM FLUSH ASYNC INSERT QUEUE`; `EXCHANGE` - drains the old table's queue itself. The guarantee is unchanged. The report of rejected - fire-and-forget inserts is a lower bound (`"complete": false`) that names the replicas it read. -- **Ownership by token.** Cloud rejects chosen table UUIDs, so helpers carry an owner token in - their `COMMENT` until the rebuild reads back and records their UUID. Do not edit helper - comments: a changed token stops the rebuild for operator reconciliation. - -Generate the rebuild from the environment's current DDL: - -```bash -ch-migrate new dev reorder_events --table events --rebuild -``` - -The command writes an `.up.sql` file under `migrations/sql/history/tables/events/` -and an irreversible revision that calls `rebuild_table`. The command prints the -exact paths; you do not edit Python. Edit only the replacement CREATE in the SQL -file, keeping the source table's name. For example: - -```sql -CREATE TABLE {db}.events ( - id UInt64, - ts DateTime, - k UInt8, - payload String -) ENGINE = MergeTree -PARTITION BY toYYYYMM(ts) -ORDER BY (k, ts, id); -``` - -Inspect and apply it: - -```bash -ch-migrate plan dev -ch-migrate up dev -``` - -The operation generates its own helper names, UUIDs and replication paths. `down` refuses this revision; reverse it with another forward rebuild migration. If live DDL cannot be fetched, `new` writes a **commented placeholder**, not an executable empty table. Supply one complete CREATE TABLE before running `up`. - -Advanced Python migrations can call `op.rebuild_table("events", "events-rebuild.sql")` or import `rebuild_table` from `ch_migrate`. -For a row transformation, `select=` is a projection expression list in target -column order, not a full SELECT query. It must preserve partition identity. -The default copies explicit insertable target columns, excluding MATERIALIZED -and ALIAS columns. Internal steps are guarded by durable state and do not need -idempotency waivers. `plan` reads the replacement SQL and runs the same preflight. - -Before starting, stop mutations and partition DDL against the source for the -whole rebuild. Ordinary inserts continue. Sharded/Distributed sources, -partition-key changes, unfinished mutations, and incompatible physical transfers -are refused. Budget roughly **2–3× the table's disk space**: the snapshot pins old -parts and the replacement stores a second copy. Helpers stay in the same database. -The snapshot and stage helpers are plain `MergeTree` without TTL, in the source's -replication family, so their merges keep every row they hold, even after a server -restart. Sorting key, partition key and physical columns match across each -transfer. An intentional engine change happens through row copy, not mismatched -ATTACH/MOVE. -Keeper helper roots are siblings of the original root, not children that could be -removed with it. Name-dependent path macros are resolved before the swap. -For Replicated tables in an Atomic database, configure a cluster covering every -source replica. An incomplete replica scope is refused rather than rebuilt on -only the connected node. Existing per-replica table UUIDs are retained and checked -individually. - -For a Replicated database, helper CREATE statements set -`database_replicated_allow_explicit_uuid=1` and -`database_replicated_allow_replicated_engine_arguments=1` for that query only. -These [documented settings](https://clickhouse.com/docs/reference/settings/session-settings/database-replicated) -preserve the journaled UUIDs and isolated Keeper paths; value `2` would replace -them and is not used. A settings profile that forbids these overrides must be -reconciled before the rebuild. The tool does not change server/profile defaults. - -### Dependent views and dictionaries - -The rebuild inspects dependents in the same database before creating its copying -view. It creates owned empty replacement helpers, then uses `EXPLAIN` and -zero-row result headers to validate source views, materialized views, dictionary -projections, and views that write `TO` the table against the replacement. -Compatible server-side conversions remain allowed; these checks cannot predict -invalid values that a future writer might supply. - -A missing/renamed column or incompatible view expression names the failing -dependent and refuses before any copying or swap. The tool removes its fresh -validation helpers and releases its own lock; the source remains unchanged. -Repairing the replacement SQL and rerunning starts a fresh attempt from that -known rejection. A failure after copying begins does not use this cleanup path: -it retains ownership/state for operator reconciliation. - -Materialized views follow the original table name across the swap, including -views with an inner engine and views that write `TO` the rebuilt table. The tool -does not re-point or recreate application views. Snapshot/staging copies do not -replay the source view's historical aggregate: a source-to-aggregate difference -can therefore equal the duplicate rows inside the recorded rebuild window. - -After the swap, the tool reloads each discovered dependent ClickHouse dictionary. -Reload errors name the dictionary and fail the migration rather than declaring -completion with an old cache. This needs the global `SYSTEM RELOAD DICTIONARY` -privilege; bootstrap grants it only where the administrator's current grants -allow it, while the runtime reloads only the discovered names. - -Distributed routes to the source are conservatively refused **even when the -route has only one shard**. Cross-database views and dictionaries that the scoped -dependency inspection cannot see are a known limitation: inspect and manage them -explicitly before rebuilding. Do not make concurrent schema/dependency changes -while a rebuild is running. - -### Recover an interrupted rebuild - -Progress is stored in the database, not on the runner's filesystem. A completed -partition is skipped; an incomplete copy is cancelled before its stage is reset. -If the swap already happened, recovery finishes cleanup rather than swapping again. - -**An existing lock always refuses another runner—even when its heartbeat expires.** -ClickHouse cannot fence a paused runner's later `EXCHANGE` using a client-side lease. -Do not delete a lock merely because it looks stale. - -1. Establish that the previous runner has stopped and cannot resume, including - any suspended process or automated restart. -2. Reconcile its running copy/DDL queries, the `_ch_migrate_journal` entry, helper - tables, and source/replacement UUIDs on every replica. Do not stamp the revision - or delete its journal to suppress an unknown-outcome error. -3. Explicitly release `_chm_rebuild_lock_` only after that reconciliation. - On a single server, for example: - `DROP TABLE analytics._chm_rebuild_lock_events SYNC`. - For a configured cluster, release on every host with the matching `ON CLUSTER`; - for a Replicated database, wait for its replicated DROP to converge. -4. Run `ch-migrate up ENV` again. It acquires a new lock and resumes the recorded - work, killing any recorded orphan copy before resetting the staging partition. - -### Recover rows stranded at the swap - -A stop that names rows reaching only `
__chm_old_` happens after the -swap: the live table is already the rebuilt one. The query IDs are those in -`system.part_log`. For an asynchronous insert this is the flush query; find the -writers' own query IDs through `flush_query_id` in `system.asynchronous_insert_log`. - -1. Copy the stranded rows into the live table, for example by selecting them from - the rollback table by key and excluding keys the live table already has. -2. Release `_chm_rebuild_lock_
` as in step 3 above. -3. Run `ch-migrate up ENV` again. The run treats the release as your - acknowledgement, finishes, and records `"stranded"` with `"acknowledged": true` - in the completion report. - -### Retain and remove the rollback table - -After the all-replica swap and copying-view barrier, the original table is retained -as `
__chm_old_`. Keep it for at least **seven days**, or your -organization's longer recovery window; there is no automatic deletion job. -Snapshot, staging, copying-view, and successful-run lock helpers are removed. - -The retained table is not a continuously updated backup. Before rolling back, -quiesce writers and reconcile rows accepted after the swap; blindly exchanging it -back can remove those newer rows from the live table. After verifying the new -table and completing the retention window, explicitly DROP the retained table, -using the appropriate replicated/cluster scope. - -The completion report includes the duplicate window, old/new UUIDs, rollback -table, old-UUID async rejection count, bytes, and query IDs, and the stranded-row -check (`"stranded"`: rows, query IDs, whether an operator acknowledged them). On ClickHouse 26.3, -the old UUID is recorded in the error-741 exception text—not a `table_uuid` log -column. Reporting drains the old storage UUID through the retained rollback -table before flushing logs and applying flush-time bounds. Flushing only the -new live name would miss the old queue. Missing inspection privileges or log -evidence are errors, not invented zero-error results. - -## Drift detection in CI - -After applying and verifying migrations in an environment, capture its expected -schema: - -```bash -ch-migrate snapshot dev -ch-migrate diff dev --snapshot-dir migrations/sql/snapshots/20261002_120000 --json -``` - -Replace the example timestamp with the directory reported by `snapshot`. Review -and commit that directory alongside the migration head. Do not regenerate the -snapshot automatically in the drift job: doing so would accept an out-of-band -change as the new expected schema. Re-snapshot after an intentional migration or -an explicitly reviewed reconciliation, not merely to turn a failed check green. - -Copy [the GitHub Actions example](docs/examples/github-actions-drift.yml) into the -consumer repository's `.github/workflows/`. It runs on a daily schedule and -same-repository pull requests; fork PRs are excluded because they have no database -secret. It does not use `pull_request_target`. All actions are pinned by commit. -The checkout must contain `config.yaml` and the committed snapshot, and the runner -must be able to reach the selected server. - -Configure these repository variables: - -| Variable | Value | -|---|---| -| `CH_MIGRATE_PACKAGE` | An approved, pinned `ch-migrate-cli` requirement that supports `diff --json`. There is no fallback to an older release. | -| `CH_MIGRATE_ENV` | An environment name in `config.yaml`, such as `dev`. | -| `CH_MIGRATE_SNAPSHOT_DIR` | That environment's committed snapshot directory. | - -Set the repository secret `DRIFT_MIGRATION_PASSWORD` to the password for that -environment's configured user. Prefer a dedicated schema-inspection account, -not an admin account. The job writes a private `.env.local` without printing the -value and removes the file even when comparison fails. - -`diff --json` exits **0** for matching schemas, **1** for drift, and **2** for a -configuration/connection error. The workflow saves `drift.json`, uploads it even -on comparison failure, then fails the job for either nonzero result. The report -names changed tables and fields. The example's install and shell steps are tested -locally against the source checkout; that does not publish the package or prove -that a previously released version contains JSON support. - -Without `--snapshot-dir`, `diff` sorts entries in `migrations/sql/snapshots/` by -name and chooses the last one. It does not select a snapshot by environment or -recorded migration revision. Use an explicit path in CI, especially when one -repository holds snapshots for several environments. - -## Statement classification - -`ch_migrate.classify.classify(statement, live_schema=None)` accepts SQL or -an extracted migration statement. It returns `Classification(kind, table, detail)`; -`live_schema` is the existing introspection `Schema`. Classification does not run -SQL or change lint's gate rules. - -- `metadata`: no background mutation, such as ADD COLUMN or a default-only change. -- `mutation`: a background mutation of parts, including DROP INDEX/PROJECTION and - RENAME COLUMN, not just UPDATE/DELETE. -- `rebuild`: the requested key or engine change cannot be made by that in-place ALTER. -- `other`: non-schema operations or work not covered by the classifier. - -Without a live column type, MODIFY COLUMN is conservatively a mutation **if the -type changes**. MODIFY TTL materializes existing parts unless the statement sets -`materialize_ttl_after_modify = 0`. Lightweight DELETE creates a mutation; -lightweight UPDATE writes synchronous patch parts and is explicitly `other`, -not a nonexistent background mutation. A sorting-key extension is metadata only -when it preserves the old key and appends columns added by the same ALTER. - -The [classification corpus](tests/corpus/classification.yaml) is checked against -fresh seeded MergeTree tables: every ALTER must create the predicted mutation, -create none, or be rejected as an unsupported in-place change. The server proof, -not the apparent SQL verb, defines the classification. - -## Plan pending migrations - -Run `ch-migrate plan dev` before `up`. It reads pending upgrade statements in -apply order, classifies them against the live schema, reports rewrite bytes and -dependent materialized views/dictionaries, and includes every lint finding. -It does not import revision modules, execute SQL from migrations, or run -`SYSTEM SYNC`. Python-generated SQL that cannot be extracted statically is not -included. The result describes current live state, not a simulation of each -preceding pending statement. - -```bash -ch-migrate plan dev -ch-migrate plan dev --json -``` - -For example, these two pending statements produce different size labels: - -```sql -ALTER TABLE analytics.events MODIFY COLUMN value UInt32; --- ch-migrate: allow-non-idempotent reviewed one-time correction -ALTER TABLE analytics.events UPDATE value = value + 1 WHERE id = 1; -``` - -- The column change reports **exact** compressed/uncompressed column bytes and - distinct active parts from `system.parts_columns` when column sizes are available. -- The UPDATE reports **ceiling: up to** the active table's compressed/uncompressed - bytes and part count. An explicit partition ID or a simple equality on - `_partition_id`, an unsigned identity partition key, or a supported date-bucket - partition expression narrows the ceiling. Other predicates remain table-wide. -- Compact parts share a data file and report zero per-column byte counters. - These use a whole-part **ceiling**, not an exact zero-byte claim. -- The waiver appears with its reason. It removes the UPDATE's gate blocker, not - its size or dependency warnings. - -Rebuild-required statements also run shared read-only preflight. It reports -Distributed/sharded layouts, changed partition keys, unfinished mutations, -incompatible physical-transfer definitions, and unacknowledged async writers. -Storage and part counts use the maximum across inspected replicas, not the sum -of replica copies. Disk-space and partition-part warnings identify each host. -The insert rate comes from completed initial INSERTs in the last 60 minutes. - -Writer inspection includes rotated `query_log_N` tables and inherited -user/profile/role settings, with a bounded log scan. Missing privileges, absent -logs, or a scan exceeding its bound are errors, not evidence of safe writers. -Unknown effective settings refuse a rebuild. A quiet or unflushed log does not -prove that no writers exist. The shared `RebuildRequest` API accepts -`allow_unacknowledged_async_loss=True` only for an explicitly reviewed -migration-file opt-in; `plan` has no command-line bypass. - -Only ReplacingMergeTree variants can collapse identical sorting-key copies on -merge. Other targets may retain duplicates, including Collapsing and -VersionedCollapsing tables with identical positive rows. - -Rebuild-classified statements include an exact suggested -`ch-migrate new ENV rebuild_TABLE --table TABLE --rebuild` command. Human output -shows the command; JSON adds the optional `suggested_command` field without -changing schema version 1. - -The [plan JSON schema](docs/schemas/plan.schema.json) covers both success and -error documents. Human and JSON output use the same facts. Exit codes are **0** -for a successful inspection, including warnings/preflight refusals; **1** when -the idempotency/standalone-SET gate would refuse `up`; and **2** for an inspection -error. Preflight refusals remain explicit in each statement's rebuild findings. -Read-only plans do not synchronize replicas or protect against concurrent changes. - ## Command reference Every command accepts `--help`. Top-level `ch-migrate --version` reports the installed package version. `ENV` below names an entry in `config.yaml`. Output lines start with `→` for a step, `✓` for a result, `!` for a warning and `✗` for an error; warnings and errors go to stderr. Colour is dropped when output is not a terminal or `NO_COLOR` is set, and lines are never wrapped, so paths and SQL can be copied or grepped. -`status`, `history`, `plan`, `lint`, and `diff` accept `--json`. JSON mode writes one document to stdout; diagnostics go to stderr. Every document includes `"schema_version": 1` and `"command"`. From 1.0 this is a public interface: breaking changes require a major version. Errors include an `error` string rather than inventing a successful empty result. The schemas linked below use JSON Schema draft 2020-12. - ### `init` `ch-migrate init [PATH] [-n NAME]` initializes the current directory by default. `-n/--name` sets the project name; otherwise it uses the directory name. @@ -828,7 +493,7 @@ Example: `ch-migrate bootstrap dev --dry-run` ### `new` -`ch-migrate new ENV NAME [--table T | --view V | --dict D] [--irreversible REASON | --python | --exchange | --rebuild]` creates SQL-first migrations by default. Object-option aliases are `-t`, `-v`, and `-d`. `--python` keeps the Python template. `--rebuild` and deprecated `--exchange` require `--table`; each conflicts with the other authoring-mode options. Conflicts fail before a revision is written. +`ch-migrate new ENV NAME [--table T | --view V | --dict D] [--irreversible REASON | --python | --exchange]` creates SQL-first migrations by default. Object-option aliases are `-t`, `-v`, and `-d`. `--python` keeps the Python template. `--exchange` requires `--table`. The three authoring-mode options are mutually exclusive, and conflicts fail before a revision is written. Example: `ch-migrate new dev add_status --table logs` @@ -846,59 +511,22 @@ Example: `ch-migrate down dev --revision base` ### `status` -`ch-migrate status ENV [--json]` shows connection information, applied/pending counts, and head status, and names the `up` command when migrations are pending. Status is a report: human output exits 0 when the database is unreachable (with a warning) or migrations are pending, and exits 1 only when the configuration or `migrations/versions/` is missing, so CI can run it as a non-blocking check. JSON mode uses nonzero exits for unreachable or pending state. +`ch-migrate status ENV` shows connection information, applied/pending counts, and head status, and names the `up` command when migrations are pending. No command-specific options. Status is a report: it exits 0 when the database is unreachable (with a warning) or migrations are pending, and exits 1 only when the configuration or `migrations/versions/` is missing, so CI can run it as a non-blocking check. Example: `ch-migrate status dev` -With `--json`, [the status schema](docs/schemas/status.schema.json) includes -`database`, `current_heads`, `script_heads`, `pending`, `applied`, and `at_head`. -`applied` includes ancestors resolved through the revision graph, not just stored -heads. JSON exits 0 when the head sets match exactly, 1 when pending or diverged, -and 2 when configuration or database state cannot be read. Human-mode exits are -unchanged. - -```bash -ch-migrate status dev --json | jq -e .current_heads -# Check the actual head set, not the last element of applied: -ch-migrate status dev --json | jq -e '.current_heads == ["PINNED_REVISION"]' -``` - -Replace `PINNED_REVISION` with the required revision. In a shell pipeline, enable -`set -o pipefail` if the caller must also preserve ch-migrate's exit code. - ### `history` -`ch-migrate history ENV [--json]` displays the revision graph and applied state. +`ch-migrate history ENV` displays the revision graph and applied state. No command-specific options. Example: `ch-migrate history dev` -`ch-migrate history dev --json` follows [the history schema](docs/schemas/history.schema.json). -Each entry in `revisions` includes its revision, all `down_revisions` (including -merge parents), description, create date, path, applied status, and irreversible -reason. A reversible revision has `irreversible: null`. If database state is -unavailable, `applied` is null, the document includes `error`, and JSON exits 2. -An unknown database head also makes applied status unknown instead of claiming -that every local revision is unapplied. - -### `plan` - -`ch-migrate plan ENV [--json]` inspects pending upgrades without executing them. -See [Plan pending migrations](#plan-pending-migrations) for byte precision, -rebuild checks, limitations, exit codes, and an annotated example. - ### `lint` -`ch-migrate lint [ENV] [--json]` analyzes upgrade statements, not downgrade SQL. Without `ENV`, it checks revisions after the gate baseline statically without credentials or a connection. With an environment, it checks only pending revisions in that scope and adds live dependency checks. Use `plan` for rewrite sizes. If it cannot determine the pending set, it fails rather than silently checking a different scope. Errors exit nonzero; warnings and waiver INFO lines alone do not. +`ch-migrate lint [ENV]` analyzes upgrade statements, not downgrade SQL. Without `ENV`, it checks revisions after the gate baseline statically without credentials or a connection. With an environment, it checks only pending revisions in that scope and adds live size and dependency checks. If it cannot determine the pending set, it fails rather than silently checking a different scope. No command-specific options. Errors exit nonzero; warnings and waiver INFO lines alone do not. Example: `ch-migrate lint` -`ch-migrate lint --json` follows [the lint schema](docs/schemas/lint.schema.json). -`findings` include rule, severity, source file/line, message, `blocking`, and -`waived`. `blocking` means the finding currently blocks `up`; a waived finding -has its written reason and does not block. `counts` reports total, error, warning, -info, blocking, and waived findings. JSON and human output use the same exit -codes: 1 for lint/configuration errors, otherwise 0. - Findings name the project-relative SQL file and statement line. Inline Python SQL points to its `op.execute` call. Extraction reads `run_sql`/`read_sql` file references and literal or f-string `op.execute` arguments without importing @@ -915,16 +543,10 @@ Example: `ch-migrate deps dev --validate migrations/sql/history/tables/logs/chan ### `diff` -`ch-migrate diff ENV [-s PATH] [--json]` compares the live schema with the latest snapshot. `-s/--snapshot-dir PATH` chooses another snapshot. Exit code 0 means no drift; 1 means drift. JSON errors exit 2; human-mode execution errors retain exit 1. +`ch-migrate diff ENV [-s PATH]` compares the live schema with the latest snapshot. `-s/--snapshot-dir PATH` chooses another snapshot. Exit code 0 means no drift; 1 means drift or an execution error. Example: `ch-migrate diff dev --snapshot-dir migrations/sql/snapshots/20261002_120000` -`ch-migrate diff dev --json` follows [the diff schema](docs/schemas/diff.schema.json). -`objects` identifies each object's type, name, status, and structural `details` -(field, local value, remote value, and message). `in_sync` is true only when every -object matches. Use the exit code as the drift gate; an error document does not -contain a fabricated `in_sync: true`. - ### `snapshot` `ch-migrate snapshot ENV [-e GLOB] [-f GLOB]` writes CREATE statements to a timestamped snapshot directory. `-e/--exclude` and `-f/--filter` accept repeated glob patterns for excluded and included objects. @@ -1015,6 +637,7 @@ Non-gate rule severities can be `error`, `warn`, or `off`. Gate rules must remai ```yaml lint: + large_table_threshold: 100000000 mv_validation_cutoff: "2026-01-01" rules: destructive_changes: warn @@ -1024,24 +647,10 @@ lint: Review findings rather than treating a successful command as a guarantee that a migration is safe. DDL and mutations are not transactional. -`large_table_mutation` and `large_table_threshold` are retired. Existing entries -are ignored with one deprecation warning pointing to `ch-migrate plan`. - ### Bootstrap roles and Cloud notes Bootstrap creates `{project}_migration_role` for schema/data operations and introspection, including explicit `system.grants`, `system.databases`, `system.tables`, and `system.mutations` access. It grants available cluster, remote-read, and synchronization rights through `CURRENT GRANTS`. Optional users add `{project}_readonly_role` (SELECT/SHOW) and `{project}_dict_role` (dictionary sources). Bootstrap uses explicit grants rather than `GRANT ALL` for Cloud compatibility. -Plan also needs readable `system.parts`, `system.parts_columns`, and, for rebuild -preflight, disks, MergeTree settings, user/profile/role metadata, and query logs. -Bootstrap grants these where the admin's current grants permit. Existing -deployments may need to rerun bootstrap or grant missing access explicitly, -including readable rotated query logs. A denied inspection fails rather than -silently omitting evidence. -Queued async inserts whose logged settings are missing are matched to active -`system.processes` entries by host and query ID. If neither source proves the -writer's acknowledgement settings, preflight still refuses without the -unacknowledged-loss opt-in. - Use standard table engine names such as `MergeTree` and `ReplacingMergeTree`; ClickHouse Cloud supplies its shared variants. Cloud usually uses HTTPS port `8443`; local HTTP usually uses `8123`. ## Development @@ -1107,44 +716,13 @@ uv run --locked pytest -q -m "integration and cloud" # Cloud-only checks Tests marked `cloud` run only when the URL is a Cloud service (Shared databases); they check that a migration's `SET` reaches every later statement -whichever replica serves it, version-table round trips on SharedMergeTree, and -mutation waiting. Cluster tests skip. Cloud grants no user `SYSTEM STOP MERGES`, -so tests that hold a mutation pending skip with that reason; the self-hosted -suite still covers them. Each test drops its own database. After an interrupted -run, `scripts/cloud_sweep.py [--dry-run]` drops `it_*` databases no query has +whichever replica serves it, and version-table round trips on SharedMergeTree. +Cluster tests skip. Cloud grants no user `SYSTEM STOP MERGES`, so tests that +hold a mutation pending skip with that reason; the self-hosted suite still +covers them. Each test drops its own database. After an interrupted run, +`scripts/cloud_sweep.py [--dry-run]` drops `it_*` databases no query has touched for 24 hours, with their test users and roles. -### Continuous-writer rebuild acceptance - -Run all five local variants five times, stopping on the first failure: - -```bash -for run in 1 2 3 4 5; do - echo "Continuous-insert run $run" - env -u CH_MIGRATE_IT_URL uv run --locked pytest -q -s \ - -m integration -k continuous_insert || exit 1 -done -``` - -The variants cover synchronous writes, process termination and operator-confirmed -resume, writes through node 1 checked on both replicas, and async writes with -`wait_for_async_insert` set to either `1` or `0`. Each writer spans three -partitions and continues after the swap. Real HTTP barriers hold snapshot and -exchange requests; they do not replace ClickHouse operations. - -Each `CONTINUOUS_INSERT` output line reports sent IDs, duplicate IDs, lost IDs, -and error-741 retries. Acknowledged writes must have no lost IDs. Duplicate -attribution uses server query intervals for synchronous inserts and actual -`asynchronous_insert_log` flush timestamps for async inserts. The fire-and-forget -variant first proves refusal without opt-in, then requires the missing IDs to -match the rebuild's old-UUID FlushError report exactly. The recovery variant -proves an expired lock still refuses takeover and that resume skips the already -moved partition after operator reconciliation. - -These tests use only owned containers, not `CH_MIGRATE_IT_URL`. CI includes -them through the existing integration markers; the replicated variant runs in -the cluster job. - ## License MIT License — see [LICENSE](LICENSE). diff --git a/ch_migrate/__init__.py b/ch_migrate/__init__.py index a07ca88..e01ef59 100644 --- a/ch_migrate/__init__.py +++ b/ch_migrate/__init__.py @@ -5,7 +5,7 @@ from ch_migrate import read_sql, get_db, get_env_config, create_dictionary """ -__version__ = "0.9.0" +__version__ = "0.7.0" from typing import Any @@ -33,10 +33,6 @@ def __getattr__(name: str) -> Any: from ch_migrate.sql import run_sql return run_sql - elif name == "rebuild_table": - from ch_migrate.rebuild import rebuild_table - - return rebuild_table elif name == "IrreversibleMigration": from ch_migrate.downgrade import IrreversibleMigration @@ -60,7 +56,6 @@ def __getattr__(name: str) -> Any: "__version__", "read_sql", "run_sql", - "rebuild_table", "IrreversibleMigration", "get_db", "get_env_config", diff --git a/ch_migrate/alembic_env.py b/ch_migrate/alembic_env.py index 1a0c860..8d55bff 100644 --- a/ch_migrate/alembic_env.py +++ b/ch_migrate/alembic_env.py @@ -4,19 +4,22 @@ import logging import os +from functools import partial from logging.config import fileConfig from pathlib import Path +from types import SimpleNamespace from typing import Any +from uuid import uuid4 -import clickhouse_connect.cc_sqlalchemy.alembic # noqa: F401 - registers official dialect +import clickhouse_connect.cc_sqlalchemy.alembic # Registers the official implementation. from alembic import context from clickhouse_connect.cc_sqlalchemy.alembic.impl import ClickHouseImpl from clickhouse_connect.driver.httputil import get_pool_manager from dotenv import load_dotenv -from sqlalchemy import URL, Column, MetaData, String, Table, create_engine, pool -from sqlalchemy.sql.dml import Delete, Insert, Update +from sqlalchemy import URL, Column, MetaData, String, Table, create_engine, pool, text +from sqlalchemy.sql.dml import Insert -import ch_migrate.rebuild # noqa: F401 - registers guarded Alembic operation +from ch_migrate import ui from ch_migrate.config import get_env_config from ch_migrate.hooks import HookRegistry, run_hooks from ch_migrate.rebase import _literal_assignment @@ -28,7 +31,7 @@ sync_version_replica, ) from ch_migrate.waiting import MigrationWaiter -from ch_migrate.waiting_versions import write_unjournaled +from ch_migrate.waiting.reruns import RerunGuard ENV_VERSION = 2 DEFAULT_SESSION_TIMEOUT = 1800 @@ -59,25 +62,45 @@ def version_table_impl(self, *, version_table, version_table_schema, version_tab ) def _exec(self, construct, execution_options=None, multiparams=None, params=None): - # The official implementation owns these SQLAlchemy version constructs; - # all user SQL, including raw bind execution, is covered by cursor events. - if isinstance(construct, (Insert, Update, Delete)) and self._is_version_table_construct( - construct - ): - waiter = self.context_opts.get("ch_migrate_waiter") - if waiter is not None and waiter.revision is not None: - return waiter.versions.execute(construct, self) - if not isinstance(construct, Delete): - return write_unjournaled(construct, self) + if isinstance(construct, Insert) and self._is_version_table_construct(construct): + self._insert_version_row(construct) + return SimpleNamespace(rowcount=1) return super()._exec( construct, execution_options=execution_options, multiparams=multiparams, params=params ) + def _exec_version_update(self, construct, execution_options=None): + # The official INSERT then waited DELETE, with the INSERT written by _insert_version_row. + self._insert_version_row(construct) + where = self._compile_version_where(construct) + super()._exec( + text( + f"ALTER TABLE {self._version_table_name} DELETE WHERE {where} " + "SETTINGS mutations_sync = 2" + ) + ) + return SimpleNamespace(rowcount=1) + + def _insert_version_row(self, construct) -> None: + # Replicated and Shared MergeTree silently drop a block identical to an earlier insert + # (DRY-1447), so re-inserting a previous head after `down` left no version row at all. + # A token unique to this write keeps every intentional version row. + value = self._compile_clause(next(iter(construct._values.values()))) + token = f"'chm_version_{uuid4().hex}'" + super()._exec( + text( + f"INSERT INTO {self._version_table_name} (version_num) " + f"SETTINGS insert_deduplication_token = {token}, async_insert = 0 " + f"VALUES ({value})" + ) + ) + def run() -> None: """Run the selected environment on one official-dialect session.""" root = Path.cwd() load_dotenv(root / ".env.local") + # `ch-migrate up` runs Alembic in its own process and names the environment directly. environment = context.config.attributes.get( "ch_migrate_environment", os.environ.get("CH_ENVIRONMENT", "dev") ) @@ -89,10 +112,11 @@ def run() -> None: os.environ.pop("CH_CLUSTER", None) if context.config.config_file_name is not None: fileConfig(context.config.config_file_name, disable_existing_loggers=False) + _route_logs(context.config.attributes.get("ch_migrate_log_handler")) if context.is_offline_mode(): _run_offline(env_config) else: - _run_online(env_config) + _run_online(env_config, environment) def has_current_env(path: Path) -> bool: @@ -103,6 +127,18 @@ def has_current_env(path: Path) -> bool: ) +def _route_logs(handler: logging.Handler | None) -> None: + # `ch-migrate up` reports each migration in its own words, so Alembic's and + # ch-migrate's records go to its handler instead of alembic.ini's console handler. + if handler is None: + return + for name in ("alembic", "ch_migrate"): + named = logging.getLogger(name) + named.handlers = [handler] + named.propagate = False + named.setLevel(logging.INFO) + + def _run_offline(env_config: dict[str, Any]) -> None: context.configure( url=_url(env_config), @@ -120,26 +156,57 @@ def _run_offline(env_config: dict[str, Any]) -> None: context.run_migrations() -def _run_online(env_config: dict[str, Any]) -> None: +def _run_online(env_config: dict[str, Any], environment: str) -> None: # The session lives on whichever replica its connection reached: ClickHouse Cloud routes # each TCP connection, not each session id. A pool of its own keeps the session on one - # connection; sharing the driver's default pool let other in-process clients (rebuild - # heartbeats) open a second connection under it. A dropped connection still fails - # closed through session_check. + # connection instead of sharing the driver's default pool with other in-process clients. + # A dropped connection still fails closed through session_check. session_pool = get_pool_manager(maxsize=1) engine = create_engine( _url(env_config), poolclass=pool.NullPool, connect_args={"pool_mgr": session_pool} ) try: with engine.connect() as connection: - _run_connection(connection, env_config) + _run_connection(connection, env_config, environment) finally: engine.dispose() session_pool.clear() -def _run_connection(connection, env_config: dict[str, Any]) -> None: - # The checked session and listener registration share this failure boundary. +def _run_connection(connection, env_config: dict[str, Any], environment: str) -> None: + hooks = HookRegistry.from_config(env_config.get("hooks")) + state = _checked_session(connection, env_config) + context.configure( + connection=connection, + target_metadata=None, + version_table="alembic_version", + version_table_schema=state.database, + on_version_apply=_post_migrate(hooks, state.database), + ch_migrate_version_state=state, + ) + waiter = MigrationWaiter(connection, state, context.config.attributes.get("ch_migrate_timeout")) + guard = _rerun_guard(waiter.client, state, environment) + try: + waiter.install(context.config) + with context.begin_transaction(): + # Work an earlier run left unfinished finishes before anything is sent again. + waiter.wait(before_run=True) + run_hooks( + connection, + hooks.pre_migrate, + db=state.database, + phase="pre_migrate", + revision="all", + ) + runtime = context.get_context() + runtime._migrations_fn = guard.steps(runtime._migrations_fn) + context.run_migrations() + finally: + waiter.close() + + +def _checked_session(connection, env_config: dict[str, Any]) -> VersionTableState: + # Establish the session before requiring its existence on every later request. connection.exec_driver_sql("SELECT 1") client = connection.connection.dbapi_connection.client client.set_client_setting("session_check", 1) @@ -149,41 +216,7 @@ def _run_connection(connection, env_config: dict[str, Any]) -> None: if warning := state.warning(): logger.warning(warning) connection.dialect.ddl_compiler = VersionTableDDLCompiler - waiter = MigrationWaiter(connection, state, context.config.attributes.get("ch_migrate_timeout")) - try: - waiter.install(context.config) - context.configure( - connection=connection, - target_metadata=None, - version_table="alembic_version", - version_table_schema=state.database, - ch_migrate_waiter=waiter, - ch_migrate_version_state=state, - ch_migrate_env_config=env_config, - ) - _run_context(waiter, HookRegistry.from_config(env_config.get("hooks"))) - finally: - waiter.close() - - -def _run_context(waiter: MigrationWaiter, hooks: HookRegistry) -> None: - runtime = context.get_context() - runtime._migrations_fn = waiter.steps(runtime._migrations_fn, hooks) - with context.begin_transaction(): - if _is_upgrade(): - waiter.ensure_version_table(runtime._version) - waiter.versions.resume() - waiter.run_pre_hooks(hooks) - else: - run_hooks( - waiter.connection, - hooks.pre_migrate, - db=waiter.state.database, - phase="pre_migrate", - revision="all", - ) - context.run_migrations() - waiter.finish_run() + return state def _url(env_config: dict[str, Any]) -> URL: @@ -202,9 +235,43 @@ def _url(env_config: dict[str, Any]) -> URL: ) -def _is_upgrade() -> bool: - explicit = context.config.attributes.get("ch_migrate_command") - if explicit is not None: - return explicit == "upgrade" - command = getattr(context.config.cmd_opts, "cmd", None) - return bool(command and command[0].__name__ == "upgrade") +def _post_migrate(hooks: HookRegistry, database: str): + def after_revision(ctx, step, heads, run_args): + revision = ( + step.up_revision + if step.is_upgrade + else (step.down_revisions[0] if step.down_revisions else "unknown") + ) + run_hooks( + ctx.connection, + hooks.post_migrate, + db=database, + phase="post_migrate", + revision=revision, + ) + + return after_revision + + +def _rerun_guard(client, state: VersionTableState, environment: str) -> RerunGuard: + attributes = context.config.attributes + # `ch-migrate up` passes --rerun-interrupted directly; `down` and a direct alembic + # command pass it as -x rerun_interrupted=REV[,REV]. + named = attributes.get("ch_migrate_rerun_interrupted") or context.get_x_argument( + as_dictionary=True + ).get("rerun_interrupted", "").split(",") + guard = RerunGuard(client, state, frozenset(name.strip() for name in named if name.strip())) + if not attributes.get("ch_migrate_gate_checked"): + # A direct `alembic upgrade` hasn't been through `ch-migrate up`'s gate. + guard.before_first_upgrade = partial(_enforce_gate, environment) + return guard + + +def _enforce_gate(environment: str) -> None: + from ch_migrate.gate import enforce_up_gate # lint imports this module + + try: + enforce_up_gate(Path.cwd(), environment) + except ui.Failure as failure: + failure.show() + raise SystemExit(1) from None diff --git a/ch_migrate/authoring.py b/ch_migrate/authoring.py index e106705..a911de1 100644 --- a/ch_migrate/authoring.py +++ b/ch_migrate/authoring.py @@ -18,9 +18,6 @@ OBJECT_DIRS = {"table": "tables", "view": "views", "dictionary": "dictionaries"} OTHER_DIR = "other" -REBUILD_REASON = ( - "Reverse a rebuild with another forward rebuild migration; automatic downgrade is unsupported." -) @dataclass(frozen=True) @@ -33,11 +30,6 @@ class NewOptions: exchange: bool = False python_migration: bool = False irreversible_reason: str | None = None - rebuild: bool = False - - @property - def irreversible(self) -> str | None: - return REBUILD_REASON if self.rebuild else self.irreversible_reason def named_objects(self) -> list[tuple[str, str]]: return [ @@ -108,7 +100,7 @@ def write_sql_files(sql_root: Path, header: RevisionHeader, options: NewOptions) upgrade = rel_dir / f"{stem}.up.sql" (sql_root / upgrade).write_text(_sql_file_text(header, "upgrade")) - if options.irreversible is not None: + if options.irreversible_reason is not None: return SqlFiles(upgrade=str(upgrade), downgrade=None) downgrade = rel_dir / f"{stem}.down.sql" @@ -116,19 +108,9 @@ def write_sql_files(sql_root: Path, header: RevisionHeader, options: NewOptions) return SqlFiles(upgrade=str(upgrade), downgrade=str(downgrade)) -def render_revision(header: RevisionHeader, files: SqlFiles, options: NewOptions) -> str: - """The revision file for a SQL-first migration or guarded rebuild.""" - irreversible = options.irreversible - operation = "rebuild_table" if options.rebuild else "run_sql" - imports = f"IrreversibleMigration, {operation}" if irreversible else operation - upgrade_call = ( - f"rebuild_table(\n {options.table_name!r},\n {files.upgrade!r},\n" - " # ClickHouse Cloud only: set True once every writer to this table uses this\n" - " # service. ch-migrate cannot see inserts made through other services.\n" - " writers_on_this_service_only=False,\n )" - if options.rebuild - else f"run_sql({files.upgrade!r})" - ) +def render_revision(header: RevisionHeader, files: SqlFiles, irreversible: str | None) -> str: + """The revision file for a SQL-first migration.""" + imports = "IrreversibleMigration, run_sql" if irreversible else "run_sql" marker = "" if irreversible: marker = ( @@ -149,7 +131,7 @@ def render_revision(header: RevisionHeader, files: SqlFiles, options: NewOptions f"depends_on = {header.depends_on!r}\n" f"{marker}\n\n" "def upgrade() -> None:\n" - f" {upgrade_call}\n\n\n" + f" run_sql({files.upgrade!r})\n\n\n" "def downgrade() -> None:\n" f"{downgrade_body}\n" ) diff --git a/ch_migrate/bootstrap.py b/ch_migrate/bootstrap.py index f177946..5954db7 100644 --- a/ch_migrate/bootstrap.py +++ b/ch_migrate/bootstrap.py @@ -147,29 +147,8 @@ def build_bootstrap_sql( f"GRANT SELECT ON system.databases TO {project}_migration_role;", f"GRANT SELECT ON system.tables TO {project}_migration_role;", f"GRANT SELECT ON system.mutations TO {project}_migration_role;", - f"GRANT SELECT ON system.columns TO {project}_migration_role;", f"GRANT CURRENT GRANTS(SELECT ON system.replicas) TO {project}_migration_role;", - f"GRANT CURRENT GRANTS(SELECT ON system.clusters) TO {project}_migration_role;", - f"GRANT CURRENT GRANTS(SELECT ON system.distributed_ddl_queue) TO {project}_migration_role;", f"GRANT CURRENT GRANTS(SELECT ON system.parts) TO {project}_migration_role;", - f"GRANT CURRENT GRANTS(SELECT ON system.parts_columns) TO {project}_migration_role;", - f"GRANT CURRENT GRANTS(SELECT ON system.disks) TO {project}_migration_role;", - f"GRANT CURRENT GRANTS(SELECT ON system.merge_tree_settings) TO {project}_migration_role;", - f"GRANT CURRENT GRANTS(SELECT ON system.settings) TO {project}_migration_role;", - f"GRANT CURRENT GRANTS(SELECT ON system.settings_profiles) TO {project}_migration_role;", - f"GRANT CURRENT GRANTS(SELECT ON system.settings_profile_elements) TO {project}_migration_role;", - f"GRANT CURRENT GRANTS(SELECT ON system.users) TO {project}_migration_role;", - f"GRANT CURRENT GRANTS(SELECT ON system.role_grants) TO {project}_migration_role;", - f"GRANT CURRENT GRANTS(SELECT ON system.query_log) TO {project}_migration_role;", - f"GRANT CURRENT GRANTS(SELECT ON system.asynchronous_inserts) TO {project}_migration_role;", - f"GRANT CURRENT GRANTS(SELECT ON system.asynchronous_insert_log) TO {project}_migration_role;", - f"GRANT CURRENT GRANTS(SELECT ON system.part_log) TO {project}_migration_role;", - f"GRANT CURRENT GRANTS(SELECT ON system.processes) TO {project}_migration_role;", - f"GRANT CURRENT GRANTS(KILL QUERY ON *.*) TO {project}_migration_role;", - f"GRANT CURRENT GRANTS(SYSTEM FLUSH LOGS ON *.*) TO {project}_migration_role;", - f"GRANT CURRENT GRANTS(SYSTEM RELOAD DICTIONARY ON *.*) TO {project}_migration_role;", - f"GRANT CURRENT GRANTS(SYSTEM FLUSH ASYNC INSERT QUEUE ON *.*) TO {project}_migration_role;", - f"GRANT CURRENT GRANTS(SYSTEM MERGES ON {db}.*) TO {project}_migration_role;", f"GRANT CURRENT GRANTS(SYSTEM SYNC REPLICA ON {db}.*) TO {project}_migration_role;", f"GRANT CURRENT GRANTS(SYSTEM SYNC DATABASE REPLICA ON {db}.*) TO {project}_migration_role;", f"GRANT CURRENT GRANTS(CLUSTER ON *.*) TO {project}_migration_role;", diff --git a/ch_migrate/classify.py b/ch_migrate/classify.py deleted file mode 100644 index 6be4a31..0000000 --- a/ch_migrate/classify.py +++ /dev/null @@ -1,304 +0,0 @@ -"""Classify ClickHouse work without executing it; the server corpus is authoritative.""" - -from __future__ import annotations - -import re -from dataclasses import dataclass -from typing import Literal - -from ch_migrate.idempotency import _ACTIONS, _action_clauses -from ch_migrate.introspect import Schema, TableDefinition -from ch_migrate.sql import _skip_quoted_or_comment -from ch_migrate.statements import MigrationStatement - -_WORD_OR_NUMBER = re.compile(r"[A-Za-z_][A-Za-z_0-9]*|\d+(?:\.\d+)?") -_TYPE_MODIFIERS = {"DEFAULT", "MATERIALIZED", "ALIAS", "CODEC", "COMMENT", "REMOVE", "TTL"} - - -@dataclass(frozen=True) -class Classification: - kind: Literal["metadata", "mutation", "rebuild", "other"] - table: str | None - detail: str - - -def classify( - statement: str | MigrationStatement, live_schema: Schema | None = None -) -> Classification: - """Classify metadata, background mutations, impossible ALTERs, or other work. - - The contract is tests/corpus/classification.yaml. Its real-server proof in - tests/integration/test_classification.py defines correctness, not a guessed - list of SQL verbs. With no live type to compare, MODIFY COLUMN is conservatively - a mutation *if the type changes*. Unsupported actions remain explicitly other. - """ - if isinstance(statement, MigrationStatement) and statement.rebuild is not None: - return Classification("rebuild", statement.rebuild.table, "Guarded online table rebuild") - tokens = _tokens(statement if isinstance(statement, str) else statement.sql) - if not tokens: - return Classification("other", None, "No SQL statement") - words = tuple(token.upper() for token in tokens) - if words[:2] == ("ALTER", "TABLE"): - return _classify_alter(tokens, live_schema) - table = _statement_target(tokens) - if words[0] in ("CREATE", "DROP", "RENAME"): - return Classification("metadata", table, "Object metadata operation") - if words[:2] == ("DELETE", "FROM"): - return Classification( - "mutation", table, "Lightweight DELETE rewrites the row-existence mask" - ) - if words[0] == "UPDATE": - return Classification( - "other", - table, - "Synchronous lightweight UPDATE writes patch parts; no background mutation record", - ) - return Classification("other", table, "Not a classified schema operation") - - -@dataclass(frozen=True) -class _Alter: - table: str - definition: TableDefinition | None - actions: tuple[tuple[str, ...], ...] - settings: tuple[str, ...] - - -def _classify_alter(tokens: tuple[str, ...], schema: Schema | None) -> Classification: - index = 2 - if tuple(token.upper() for token in tokens[index : index + 2]) == ("IF", "EXISTS"): - index += 2 - table, index = _name(tokens, index) - if tuple(token.upper() for token in tokens[index : index + 2]) == ("ON", "CLUSTER"): - index += 3 - actions, settings = _split_settings(tokens[index:]) - definition = _table_definition(schema, table) - alter = _Alter(table, definition, _alter_actions(actions), settings) - classifications = [_classify_action(action, alter) for action in alter.actions] - priority = {"metadata": 0, "mutation": 1, "other": 2, "rebuild": 3} - kind = max(classifications, key=lambda item: priority[item.kind]).kind - return Classification( - kind, table, "; ".join(dict.fromkeys(item.detail for item in classifications)) - ) - - -def _alter_actions(tokens: tuple[str, ...]) -> tuple[tuple[str, ...], ...]: - actions: list[tuple[str, ...]] = [] - for clause in _action_clauses(tokens): - starts_action = clause and clause[0].upper() in _ACTIONS | {"REMOVE"} - if actions and (not starts_action or clause[1:2] == ("=",)): - actions[-1] += (",",) + clause - else: - actions.append(clause) - return tuple(actions) - - -def _classify_action(action: tuple[str, ...], alter: _Alter) -> Classification: - words = tuple(token.upper() for token in action) - prefix = words[:2] - if prefix == ("MODIFY", "COLUMN"): - return _modify_column(action[2:], alter) - if words[:3] == ("MODIFY", "ORDER", "BY"): - return _modify_order(action[3:], alter) - if prefix in (("MODIFY", "PRIMARY"), ("MODIFY", "PARTITION"), ("MODIFY", "ENGINE")): - return Classification( - "rebuild", alter.table, "This key or engine cannot be changed by an in-place ALTER" - ) - if prefix == ("MODIFY", "TTL"): - disabled = _setting_is_zero(alter.settings, "materialize_ttl_after_modify") - return Classification( - "metadata" if disabled else "mutation", - alter.table, - "TTL metadata only" if disabled else "MODIFY TTL materializes existing parts", - ) - if words[:1] in (("UPDATE",), ("DELETE",), ("CLEAR",), ("MATERIALIZE",)): - return Classification("mutation", alter.table, "Background mutation of existing parts") - if prefix in ( - ("DROP", "COLUMN"), - ("RENAME", "COLUMN"), - ("DROP", "INDEX"), - ("DROP", "PROJECTION"), - ): - return Classification( - "mutation", - alter.table, - "Column, index, or projection change creates a background mutation", - ) - if words[:1] in (("ADD",), ("COMMENT",), ("RESET",)) or prefix in ( - ("MODIFY", "SETTING"), - ("REMOVE", "TTL"), - ): - return Classification("metadata", alter.table, "Metadata change; no background mutation") - return Classification( - "other", alter.table, "ALTER action is not covered by the server-backed classification" - ) - - -def _modify_column(tokens: tuple[str, ...], alter: _Alter) -> Classification: - if tuple(token.upper() for token in tokens[:2]) == ("IF", "EXISTS"): - tokens = tokens[2:] - if not tokens: - return Classification("other", alter.table, "Missing column definition") - column = _identifier(tokens[0]) - type_tokens = _column_type(tokens[1:]) - if not type_tokens: - return Classification( - "metadata", alter.table, "Column default, comment, or codec metadata only" - ) - current = ( - next((item for item in alter.definition.columns if item.name == column), None) - if alter.definition - else None - ) - if current is None: - return Classification( - "mutation", alter.table, "Rewrites existing parts if the type changes" - ) - if _normalize_type(type_tokens) == _normalize_type(_tokens(current.type)): - return Classification("metadata", alter.table, "Column type is unchanged") - return Classification( - "mutation", - alter.table, - f"Column {column} changes type from {current.type} to {''.join(type_tokens)}", - ) - - -def _modify_order(tokens: tuple[str, ...], alter: _Alter) -> Classification: - if not alter.definition: - return Classification( - "rebuild", - alter.table, - "Sorting-key change requires live schema to prove a new-column-only extension", - ) - requested = _order_expressions(tokens) - current = [_normalize_type(_tokens(item)) for item in alter.definition.order_by] - added = set() - for action in alter.actions: - if tuple(token.upper() for token in action[:2]) == ("ADD", "COLUMN"): - index = ( - 5 if tuple(token.upper() for token in action[2:5]) == ("IF", "NOT", "EXISTS") else 2 - ) - if len(action) > index: - added.add(_normalize_type((action[index],))) - existing = {_normalize_type((column.name,)) for column in alter.definition.columns} - suffix = requested[len(current) :] - if requested[: len(current)] == current and all(item in added - existing for item in suffix): - return Classification( - "metadata", - alter.table, - "Sorting key unchanged or extended only by columns added in this ALTER", - ) - return Classification( - "rebuild", alter.table, "Changing existing sorting-key expressions requires a table rebuild" - ) - - -def _statement_target(tokens: tuple[str, ...]) -> str | None: - words = tuple(token.upper() for token in tokens) - if words[0] == "UPDATE": - return _name(tokens, 1)[0] - if words[:2] in (("INSERT", "INTO"), ("DELETE", "FROM")): - return _name(tokens, 2)[0] - if words[0] not in ("CREATE", "DROP", "RENAME"): - return None - types = {"TABLE", "VIEW", "DICTIONARY", "DATABASE", "USER", "ROLE"} - index = next((index + 1 for index, token in enumerate(words) if token in types), len(tokens)) - while index < len(words) and words[index] in ("IF", "NOT", "EXISTS"): - index += 1 - return _name(tokens, index)[0] if index < len(tokens) else None - - -def _table_definition(schema: Schema | None, name: str) -> TableDefinition | None: - if schema is None: - return None - database, _, table = name.rpartition(".") - if database and schema.database and database != schema.database: - return None - return schema.tables.get(table or name) - - -def _split_settings(tokens: tuple[str, ...]) -> tuple[tuple[str, ...], tuple[str, ...]]: - depth = 0 - for index, token in enumerate(tokens): - if token in ("(", "["): - depth += 1 - elif token in (")", "]"): - depth -= 1 - elif token.upper() == "SETTINGS" and depth == 0: - return tokens[:index], tokens[index + 1 :] - return tokens, () - - -def _setting_is_zero(tokens: tuple[str, ...], name: str) -> bool: - value = None - for index in range(len(tokens) - 2): - if tokens[index].lower() == name and tokens[index + 1] == "=": - value = tokens[index + 2] - return value == "0" - - -def _column_type(tokens: tuple[str, ...]) -> tuple[str, ...]: - depth = 0 - for index, token in enumerate(tokens): - if depth == 0 and token.upper() in _TYPE_MODIFIERS: - return tokens[:index] - if token in ("(", "["): - depth += 1 - elif token in (")", "]"): - depth -= 1 - return tokens - - -def _order_expressions(tokens: tuple[str, ...]) -> list[str]: - if tokens[:1] == ("(",) and tokens[-1:] == (")",): - tokens = tokens[1:-1] - elif tokens[:2] == ("tuple", "(") and tokens[-1:] == (")",): - tokens = tokens[2:-1] - return [_normalize_type(action) for action in _action_clauses(tokens)] if tokens else [] - - -def _normalize_type(tokens: tuple[str, ...]) -> str: - return "".join(_identifier(token) for token in tokens) - - -def _name(tokens: tuple[str, ...], index: int) -> tuple[str, int]: - if index >= len(tokens): - return "", index - names = [_identifier(tokens[index])] - index += 1 - while index + 1 < len(tokens) and tokens[index] == ".": - names.append(_identifier(tokens[index + 1])) - index += 2 - return ".".join(names), index - - -def _identifier(token: str) -> str: - if token[:1] in ('"', "`") and token[-1:] == token[:1]: - quote = token[0] - return token[1:-1].replace(quote * 2, quote).replace("\\" + quote, quote) - return token - - -def _tokens(sql: str) -> tuple[str, ...]: - tokens = [] - index = 0 - while index < len(sql): - end = _skip_quoted_or_comment(sql, index) - if end is not None: - if sql[index] in "'\"`$": - tokens.append(sql[index:end]) - index = end - continue - if sql[index] == "{" and (end := sql.find("}", index + 1)) != -1: - tokens.append(sql[index : end + 1]) - index = end + 1 - continue - match = _WORD_OR_NUMBER.match(sql, index) - if match: - tokens.append(match.group()) - index = match.end() - continue - if not sql[index].isspace() and sql[index] != ";": - tokens.append(sql[index]) - index += 1 - return tuple(tokens) diff --git a/ch_migrate/cli.py b/ch_migrate/cli.py index 5f9d99c..48ef4ef 100644 --- a/ch_migrate/cli.py +++ b/ch_migrate/cli.py @@ -3,6 +3,7 @@ from __future__ import annotations import math +import os import re import shutil import subprocess @@ -16,17 +17,8 @@ from dotenv import load_dotenv from ch_migrate import __version__, ui -from ch_migrate.authoring import NewOptions, SqlFiles +from ch_migrate.authoring import NewOptions from ch_migrate.config import get_env_config -from ch_migrate.json_output import ( - JsonCommand, - command_failure, - diff_document, - emit_json, - history_document, - lint_document, - status_document, -) from ch_migrate.runner import alembic_failure, run_alembic, run_migrations if TYPE_CHECKING: @@ -51,41 +43,6 @@ def render_template(template_path: Path, **kwargs: str) -> str: return content -def _enforce_up_gate(environment: str, skip_mv_check: bool) -> None: - from ch_migrate.gate import lint_pending_up - - try: - report = lint_pending_up(Path.cwd(), environment, skip_mv_check) - except Exception as error: - raise ui.Failure(f"Migration preflight failed: {error}") from error - if _print_gate_findings(report): - raise ui.Failure( - "Migration gate failed; nothing was applied.", - "Fix SQL or add a reasoned in-file waiver.", - ) - - -def _print_gate_findings(report) -> bool: - from ch_migrate.display import lint_finding_line - from ch_migrate.lint import GATE_RULES, Severity - - blocked = False - for finding in report.results: - gate_error = finding.rule in GATE_RULES and finding.severity == Severity.ERROR - blocked |= gate_error - severity = finding.severity.value - if severity == "error" and not gate_error: - severity = "warn" - position = finding.file or "config.yaml" - if finding.line is not None: - position += f":{finding.line}" - console = ui.err if gate_error else ui.out - console.print(lint_finding_line(finding, location=position, severity=severity)) - if finding.statement: - ui.detail(finding.statement, stderr=gate_error) - return blocked - - def _require_current_env() -> None: from ch_migrate.alembic_env import has_current_env @@ -282,9 +239,21 @@ def bootstrap(environment: str, dry_run: bool, verbose: bool) -> None: is_flag=True, help="Skip materialized view declaration validation", ) +@click.option( + "--rerun-interrupted", + "rerun_interrupted", + multiple=True, + metavar="REVISION", + help="Run this interrupted revision again after checking what its earlier run changed", +) @click.option("--verbose", is_flag=True, help="Show the full traceback if a migration fails") def up( - environment: str, revision: str, skip_mv_check: bool, timeout: float | None, verbose: bool + environment: str, + revision: str, + skip_mv_check: bool, + timeout: float | None, + rerun_interrupted: tuple[str, ...], + verbose: bool, ) -> None: """Apply pending migrations. @@ -293,39 +262,50 @@ def up( Idempotency and standalone-SET gate errors refuse the run before Alembic. Other findings are warnings; --skip-mv-check only skips MV declaration checks. + A revision interrupted on an earlier run reruns on its own only if every statement + in it passes the gate without a waiver; otherwise name it with --rerun-interrupted. """ if timeout is not None and not math.isfinite(timeout): raise click.BadParameter("must be finite", param_hint="--timeout") _require_current_env() - _enforce_up_gate(environment, skip_mv_check) - from ch_migrate.migration_runner import run_upgrade + from ch_migrate.gate import enforce_up_gate - try: - run_upgrade(environment, revision, timeout) - except Exception as error: - if verbose: - raise - raise ui.Failure(str(error)) from error + enforce_up_gate(Path.cwd(), environment, skip_mv_check) + from ch_migrate.migration_runner import Upgrade, run_upgrade + + sys.exit(run_upgrade(Upgrade(environment, revision, timeout, verbose, rerun_interrupted))) @main.command() @click.argument("environment") @click.option("--revision", "-r", default="-1", help="Revision to downgrade to (default: -1)") +@click.option( + "--rerun-interrupted", + "rerun_interrupted", + multiple=True, + metavar="REVISION", + help="Run this interrupted revision again after checking what its earlier run changed", +) @click.option("--verbose", is_flag=True, help="Show the full traceback if a migration fails") -def down(environment: str, revision: str, verbose: bool) -> None: +def down( + environment: str, revision: str, rerun_interrupted: tuple[str, ...], verbose: bool +) -> None: """Rollback migrations. By default, rolls back the last migration. Use --revision to specify a target. + A downgrade interrupted on an earlier run reruns on its own only if every statement + in it passes the gate without a waiver; otherwise name it with --rerun-interrupted. """ _require_current_env() _refuse_irreversible_downgrade(environment, revision) - sys.exit(run_migrations(environment, ["downgrade", revision], verbose=verbose)) + # The downgrade runs in an Alembic subprocess; -x passes the names on to env.py. + named = ["-x", f"rerun_interrupted={','.join(rerun_interrupted)}"] if rerun_interrupted else [] + sys.exit(run_migrations(environment, [*named, "downgrade", revision], verbose=verbose)) -@main.command(cls=JsonCommand) +@main.command() @click.argument("environment") -@click.option("--json", "json_output", is_flag=True, help="Emit a versioned JSON document") -def status(environment: str, json_output: bool) -> None: +def status(environment: str) -> None: """Show migration status. Displays environment info, applied/pending counts, and head status. @@ -336,12 +316,6 @@ def status(environment: str, json_output: bool) -> None: from ch_migrate.display import render_status state = _load_migration_state(environment) - if json_output: - if state.db_error is not None: - command_failure(state.db_error) - document = status_document(state.graph, state.heads, state.env_config["database"]) - emit_json("status", document) - sys.exit(0 if document["at_head"] else 1) render_status( environment, state.env_config, state.graph, state.applied, db_error=state.db_error ) @@ -354,10 +328,9 @@ def status(environment: str, json_output: bool) -> None: ui.hint(f"Run `ch-migrate up {environment}` to apply {pending} pending {noun}.") -@main.command(cls=JsonCommand) +@main.command() @click.argument("environment") -@click.option("--json", "json_output", is_flag=True, help="Emit a versioned JSON document") -def history(environment: str, json_output: bool) -> None: +def history(environment: str) -> None: """Show migration history. Displays a tree of all migrations, color-coded by applied status. @@ -366,14 +339,6 @@ def history(environment: str, json_output: bool) -> None: from ch_migrate.display import render_history state = _load_migration_state(environment) - if json_output: - document = history_document( - state.graph, None if state.heads - set(state.graph.migrations) else state.applied - ) - if state.db_error is not None: - document["error"] = state.db_error - emit_json("history", document) - sys.exit(2 if state.db_error is not None else 0) render_history(state.graph, state.applied, db_error=state.db_error) @@ -383,7 +348,6 @@ class _MigrationState: graph: RevisionGraph applied: set[str] | None # None when the database could not be read db_error: str | None - heads: set[str] def _load_migration_state(environment: str) -> _MigrationState: @@ -399,7 +363,7 @@ def _load_migration_state(environment: str) -> _MigrationState: if warning := state.version_table.warning(): ui.warn(warning) except Exception as e: - return _MigrationState(env_config, graph, None, str(e), set()) + return _MigrationState(env_config, graph, None, str(e)) applied: set[str] = set() unknown = [head for head in heads if head not in graph.migrations] for head in heads: @@ -409,23 +373,19 @@ def _load_migration_state(environment: str) -> _MigrationState: ui.warn(f"The database is at {head[:12]}, which is not in your local migration files.") if unknown: ui.warn("Applied status may be incomplete; pull the missing revisions.") - return _MigrationState(env_config, graph, applied, None, heads) + return _MigrationState(env_config, graph, applied, None) def _env_config_or_fail(environment: str) -> dict[str, Any]: try: return get_env_config(environment, Path.cwd() / "config.yaml") except Exception as e: - if click.get_current_context().params.get("json_output"): - command_failure(f"Error loading config: {e}") ui.fail(f"Could not load config: {e}") def _versions_dir_or_fail() -> Path: versions_dir = Path.cwd() / "migrations" / "versions" if not versions_dir.exists(): - if click.get_current_context().params.get("json_output"): - command_failure("Error: migrations/versions/ not found") ui.fail("migrations/versions/ not found.", "Run `ch-migrate init` to create a project.") return versions_dir @@ -470,9 +430,6 @@ def _check_mv_declarations() -> None: @click.option( "--exchange", is_flag=True, help="Generate EXCHANGE TABLES scaffold (requires --table)" ) -@click.option( - "--rebuild", is_flag=True, help="Generate a guarded online rebuild (requires --table)" -) @click.option( "--python", "python_migration", is_flag=True, help="Keep the Python migration template" ) @@ -489,7 +446,6 @@ def new( view_name: str | None, dict_name: str | None, exchange: bool, - rebuild: bool, python_migration: bool, irreversible_reason: str | None, ) -> None: @@ -497,31 +453,19 @@ def new( Name an object with --table, --view or --dict to group its SQL history. Use --irreversible REASON when a change cannot restore dropped data. - --rebuild writes a guarded replacement CREATE; --exchange is deprecated. + --python keeps the Python template; --exchange still requires --table. """ options = NewOptions( - table_name=table_name, - view_name=view_name, - dict_name=dict_name, - exchange=exchange, - python_migration=python_migration, - irreversible_reason=irreversible_reason, - rebuild=rebuild, + table_name, view_name, dict_name, exchange, python_migration, irreversible_reason ) _check_new_options(options) - if exchange: - ui.warn( - "Deprecated: --exchange uses legacy copy-and-swap and can lose rows written during the copy. Use --rebuild for guarded online rebuilding." - ) result = run_alembic(environment, ["revision", "-m", name]) if result.returncode != 0: ui.fail(f"Could not create the revision: {alembic_failure(result)}") migration_path = _find_migration_file(result.stdout) if migration_path is None: ui.fail("Could not find the revision file Alembic generated.") - if rebuild: - _create_rebuild_scaffold(environment, migration_path, options) - elif exchange: + if exchange: revision = _extract_revision_from_output(result.stdout) or "" _create_exchange_scaffold(environment, table_name or "", revision, result.stdout) elif python_migration: @@ -538,12 +482,6 @@ def _check_new_options(options: NewOptions) -> None: problems.append("--exchange requires --table") if options.exchange and options.python_migration: problems.append("--exchange cannot be combined with --python") - if options.rebuild and not options.table_name: - problems.append("--rebuild requires --table") - if options.rebuild and ( - options.exchange or options.python_migration or options.irreversible_reason is not None - ): - problems.append("--rebuild cannot be combined with --exchange, --python or --irreversible") if options.irreversible_reason is not None: if options.exchange or options.python_migration: problems.append("--irreversible cannot be combined with --python or --exchange") @@ -555,12 +493,12 @@ def _check_new_options(options: NewOptions) -> None: sys.exit(1) -def _create_sql_first_migration(migration_path: Path, options: NewOptions) -> SqlFiles: +def _create_sql_first_migration(migration_path: Path, options: NewOptions) -> None: from ch_migrate.authoring import read_revision_header, render_revision, write_sql_files header = read_revision_header(migration_path) files = write_sql_files(Path.cwd() / "migrations" / "sql", header, options) - migration_path.write_text(render_revision(header, files, options)) + migration_path.write_text(render_revision(header, files, options.irreversible_reason)) ui.success(f"Created migration {header.revision[:8]} {header.message}") ui.detail(f"migrations/sql/{files.upgrade}") if files.downgrade: @@ -569,39 +507,6 @@ def _create_sql_first_migration(migration_path: Path, options: NewOptions) -> Sq else: ui.hint("Write the SQL in this file; the revision needs no edits.") ui.hint("It is marked irreversible, so `ch-migrate down` will refuse to revert it.") - return files - - -def _create_rebuild_scaffold(environment: str, migration_path: Path, options: NewOptions) -> None: - from botocore.exceptions import BotoCoreError, ClientError - - from ch_migrate.scaffold import fetch_current_ddl, generate_rebuild_sql - from ch_migrate.secrets import SSMJsonKeyError, SSMSecretNotFoundError - - table = options.table_name or "" - current_ddl = None - try: - current_ddl = fetch_current_ddl( - get_env_config(environment, Path.cwd() / "config.yaml"), table - ) - except ( - OSError, - ValueError, - BotoCoreError, - ClientError, - SSMJsonKeyError, - SSMSecretNotFoundError, - ): - pass # Offline authoring produces a commented, non-executable placeholder. - content = generate_rebuild_sql(table, current_ddl) - files = _create_sql_first_migration(migration_path, options) - (Path.cwd() / "migrations" / "sql" / files.upgrade).write_text(content) - if current_ddl: - ui.hint( - f"Fetched current DDL for {table}; edit its ORDER BY or engine and keep the table name." - ) - else: - ui.warn("Live DDL unavailable: replace the commented placeholder before running up.") def _create_python_migration(migration_path: Path, options: NewOptions) -> None: @@ -852,39 +757,16 @@ def skill(target: str) -> None: ui.success(f"Installed the skill to {skill_dst}") -@main.command(cls=JsonCommand) -@click.argument("environment") -@click.option("--json", "json_output", is_flag=True, help="Emit a versioned JSON document") -def plan(environment: str, json_output: bool) -> None: - """Inspect pending upgrades, rewrite bytes, rebuild risks and lint without executing.""" - from ch_migrate.plan import build_plan, render_plan - - try: - document = build_plan(Path.cwd(), environment) - except Exception as error: - if json_output: - emit_json("plan", {"error": str(error)}) - else: - ui.error(str(error)) - raise click.exceptions.Exit(2) from error - if json_output: - emit_json("plan", document) - else: - render_plan(document) - raise click.exceptions.Exit(1 if document["gate_would_refuse"] else 0) - - -@main.command(cls=JsonCommand) +@main.command() @click.argument("environment", required=False, default=None) -@click.option("--json", "json_output", is_flag=True, help="Emit a versioned JSON document") -def lint(environment: str | None, json_output: bool) -> None: +def lint(environment: str | None) -> None: """Lint upgrade statements with their source file and line. Without an environment, checks revisions after the gate baseline statically, without credentials or a database connection. - With an environment, checks only pending revisions and adds live dependency - checks. Use plan for rewrite sizes. Fails if the pending scope cannot be determined. + With an environment, checks only pending revisions and adds live size and + dependency checks. Fails if the pending scope cannot be determined. \b Examples: @@ -924,8 +806,6 @@ def lint(environment: str | None, json_output: bool) -> None: ) client = get_client(env_config) except Exception as e: - if json_output: - command_failure(f"Could not work out the pending revisions for {environment}: {e}") ui.fail(f"Could not work out the pending revisions for {environment}: {e}") try: @@ -937,17 +817,12 @@ def lint(environment: str | None, json_output: bool) -> None: revisions=revisions, ) except (OSError, SyntaxError, ValueError) as e: - if json_output: - command_failure(str(e)) ui.fail(str(e)) finally: if client is not None: client.close() - if json_output: - emit_json("lint", lint_document(report)) - else: - render_lint_report(report, runtime=environment is not None) + render_lint_report(report, runtime=environment is not None) sys.exit(1 if report.has_errors else 0) @@ -1005,16 +880,15 @@ def deps(environment: str, validate_sql: str | None) -> None: ui.success(f"Validation passed: {validate_sql} keeps every dependency intact.") -@main.command(name="diff", cls=JsonCommand) +@main.command(name="diff") @click.argument("environment") -@click.option("--json", "json_output", is_flag=True, help="Emit a versioned JSON document") @click.option( "--snapshot-dir", "-s", type=click.Path(exists=True), help="Path to a snapshot directory to compare against. Defaults to latest snapshot.", ) -def diff_cmd(environment: str, snapshot_dir: str | None, json_output: bool) -> None: +def diff_cmd(environment: str, snapshot_dir: str | None) -> None: """Detect schema drift between local snapshot and live database. Compares the most recent snapshot (or a specified one) against the live @@ -1045,15 +919,10 @@ def diff_cmd(environment: str, snapshot_dir: str | None, json_output: bool) -> N snapshots_base = Path.cwd() / "migrations" / "sql" / "snapshots" dirs = sorted(snapshots_base.iterdir()) if snapshots_base.exists() else [] if not dirs: - if json_output: - command_failure( - f"No snapshots found. Run `ch-migrate snapshot {environment}` first." - ) ui.fail("No snapshots found.", f"Run `ch-migrate snapshot {environment}` first.") snap_path = dirs[-1] - if not json_output: - ui.step(f"Comparing snapshot {snap_path.name} with {environment} ({database})") + ui.step(f"Comparing snapshot {snap_path.name} with {environment} ({database})") # Load local schema from snapshot files local_schema = Schema(database=database) @@ -1102,21 +971,13 @@ def diff_cmd(environment: str, snapshot_dir: str | None, json_output: bool) -> N # Get live schema try: client = get_client(env_config) - try: - live_schema = get_live_schema(client, database) - finally: - client.close() + live_schema = get_live_schema(client, database) except Exception as e: - if json_output: - command_failure(f"Error connecting to {environment}: {e}") ui.fail(f"Could not read the live schema from {environment}: {e}") # Compare diffs = compare_schemas(local_schema, live_schema) - if json_output: - emit_json("diff", diff_document(diffs)) - else: - render_diff_report(diffs) + render_diff_report(diffs) if any(d.status != DiffStatus.IN_SYNC for d in diffs): ui.hint( @@ -1207,7 +1068,7 @@ def snapshot(environment: str, exclude: tuple[str, ...], include_filter: tuple[s from ch_migrate.connection import get_client from ch_migrate.display import render_snapshot_progress - from ch_migrate.introspect import get_live_schema + from ch_migrate.introspect import Schema, get_live_schema env_config = _env_config_or_fail(environment) database = env_config["database"] diff --git a/ch_migrate/deps.py b/ch_migrate/deps.py index c355feb..a71a071 100644 --- a/ch_migrate/deps.py +++ b/ch_migrate/deps.py @@ -67,17 +67,21 @@ def validate_migration(sql: str, graph: DependencyGraph) -> list[MigrationWarnin data_flow_deps.append(edge.target) if schema_deps: - warnings.append(MigrationWarning( - severity="error", - message=f"DROP TABLE {table_name} breaks schema dependencies: {', '.join(schema_deps)}", - affected_objects=schema_deps, - )) + warnings.append( + MigrationWarning( + severity="error", + message=f"DROP TABLE {table_name} breaks schema dependencies: {', '.join(schema_deps)}", + affected_objects=schema_deps, + ) + ) if data_flow_deps: - warnings.append(MigrationWarning( - severity="warning", - message=f"DROP TABLE {table_name} breaks data flow to: {', '.join(data_flow_deps)}", - affected_objects=data_flow_deps, - )) + warnings.append( + MigrationWarning( + severity="warning", + message=f"DROP TABLE {table_name} breaks data flow to: {', '.join(data_flow_deps)}", + affected_objects=data_flow_deps, + ) + ) # Check DROP VIEW / DROP MATERIALIZED VIEW for m in _RE_DROP_VIEW.finditer(sql): @@ -86,11 +90,13 @@ def validate_migration(sql: str, graph: DependencyGraph) -> list[MigrationWarnin affected = graph.affected_by_drop(view_name) if affected: affected_names = [n.name for n in affected] - warnings.append(MigrationWarning( - severity="warning", - message=f"DROP VIEW {view_name} affects: {', '.join(affected_names)}", - affected_objects=affected_names, - )) + warnings.append( + MigrationWarning( + severity="warning", + message=f"DROP VIEW {view_name} affects: {', '.join(affected_names)}", + affected_objects=affected_names, + ) + ) # Check DROP DICTIONARY for m in _RE_DROP_DICT.finditer(sql): @@ -99,11 +105,13 @@ def validate_migration(sql: str, graph: DependencyGraph) -> list[MigrationWarnin affected = graph.affected_by_drop(dict_name) if affected: affected_names = [n.name for n in affected] - warnings.append(MigrationWarning( - severity="warning", - message=f"DROP DICTIONARY {dict_name} affects: {', '.join(affected_names)}", - affected_objects=affected_names, - )) + warnings.append( + MigrationWarning( + severity="warning", + message=f"DROP DICTIONARY {dict_name} affects: {', '.join(affected_names)}", + affected_objects=affected_names, + ) + ) return warnings diff --git a/ch_migrate/diff.py b/ch_migrate/diff.py index 1cd7ee8..f591b4c 100644 --- a/ch_migrate/diff.py +++ b/ch_migrate/diff.py @@ -52,48 +52,60 @@ def _compare_columns( remote_col = remote_map.get(name) if local_col and not remote_col: - diffs.append(FieldDiff( - field_name=f"column '{name}'", - local_value=local_col.type, - remote_value=None, - message=f"column '{name}' {local_col.type} is in the snapshot but not in the database", - )) + diffs.append( + FieldDiff( + field_name=f"column '{name}'", + local_value=local_col.type, + remote_value=None, + message=f"column '{name}' {local_col.type} is in the snapshot but not in the database", + ) + ) elif remote_col and not local_col: - diffs.append(FieldDiff( - field_name=f"column '{name}'", - local_value=None, - remote_value=remote_col.type, - message=f"column '{name}' {remote_col.type} is in the database but not in the snapshot", - )) + diffs.append( + FieldDiff( + field_name=f"column '{name}'", + local_value=None, + remote_value=remote_col.type, + message=f"column '{name}' {remote_col.type} is in the database but not in the snapshot", + ) + ) elif local_col and remote_col: if local_col.type != remote_col.type: - diffs.append(FieldDiff( - field_name=f"column '{name}' type", - local_value=local_col.type, - remote_value=remote_col.type, - message=f"column '{name}' type differs: {local_col.type} vs {remote_col.type}", - )) + diffs.append( + FieldDiff( + field_name=f"column '{name}' type", + local_value=local_col.type, + remote_value=remote_col.type, + message=f"column '{name}' type differs: {local_col.type} vs {remote_col.type}", + ) + ) if local_col.default_kind != remote_col.default_kind: - diffs.append(FieldDiff( - field_name=f"column '{name}' default_kind", - local_value=local_col.default_kind, - remote_value=remote_col.default_kind, - message=f"column '{name}' default kind differs", - )) + diffs.append( + FieldDiff( + field_name=f"column '{name}' default_kind", + local_value=local_col.default_kind, + remote_value=remote_col.default_kind, + message=f"column '{name}' default kind differs", + ) + ) if local_col.default_expr != remote_col.default_expr: - diffs.append(FieldDiff( - field_name=f"column '{name}' default_expr", - local_value=local_col.default_expr, - remote_value=remote_col.default_expr, - message=f"column '{name}' default expression differs", - )) + diffs.append( + FieldDiff( + field_name=f"column '{name}' default_expr", + local_value=local_col.default_expr, + remote_value=remote_col.default_expr, + message=f"column '{name}' default expression differs", + ) + ) if local_col.codec != remote_col.codec: - diffs.append(FieldDiff( - field_name=f"column '{name}' codec", - local_value=local_col.codec, - remote_value=remote_col.codec, - message=f"column '{name}' codec differs: {local_col.codec} vs {remote_col.codec}", - )) + diffs.append( + FieldDiff( + field_name=f"column '{name}' codec", + local_value=local_col.codec, + remote_value=remote_col.codec, + message=f"column '{name}' codec differs: {local_col.codec} vs {remote_col.codec}", + ) + ) return diffs @@ -104,51 +116,61 @@ def _compare_tables(local: TableDefinition, remote: TableDefinition) -> list[Fie # Engine if local.engine != remote.engine: - diffs.append(FieldDiff( - field_name="engine", - local_value=local.engine, - remote_value=remote.engine, - message=f"engine differs: {local.engine} vs {remote.engine}", - )) + diffs.append( + FieldDiff( + field_name="engine", + local_value=local.engine, + remote_value=remote.engine, + message=f"engine differs: {local.engine} vs {remote.engine}", + ) + ) # Columns diffs.extend(_compare_columns(local.columns, remote.columns)) # ORDER BY if local.order_by != remote.order_by: - diffs.append(FieldDiff( - field_name="order_by", - local_value=", ".join(local.order_by), - remote_value=", ".join(remote.order_by), - message=f"ORDER BY differs", - )) + diffs.append( + FieldDiff( + field_name="order_by", + local_value=", ".join(local.order_by), + remote_value=", ".join(remote.order_by), + message=f"ORDER BY differs", + ) + ) # PARTITION BY if local.partition_by != remote.partition_by: - diffs.append(FieldDiff( - field_name="partition_by", - local_value=local.partition_by, - remote_value=remote.partition_by, - message=f"PARTITION BY differs", - )) + diffs.append( + FieldDiff( + field_name="partition_by", + local_value=local.partition_by, + remote_value=remote.partition_by, + message=f"PARTITION BY differs", + ) + ) # TTL if local.ttl != remote.ttl: - diffs.append(FieldDiff( - field_name="ttl", - local_value=local.ttl, - remote_value=remote.ttl, - message=f"TTL differs", - )) + diffs.append( + FieldDiff( + field_name="ttl", + local_value=local.ttl, + remote_value=remote.ttl, + message=f"TTL differs", + ) + ) # Settings if local.settings != remote.settings: - diffs.append(FieldDiff( - field_name="settings", - local_value=str(local.settings), - remote_value=str(remote.settings), - message=f"SETTINGS differ", - )) + diffs.append( + FieldDiff( + field_name="settings", + local_value=str(local.settings), + remote_value=str(remote.settings), + message=f"SETTINGS differ", + ) + ) return diffs @@ -156,6 +178,7 @@ def _compare_tables(local: TableDefinition, remote: TableDefinition) -> list[Fie def _normalize_ddl(raw: str) -> str: """Normalize raw DDL for fallback string comparison.""" import re + s = raw.strip() s = re.sub(r"\s+", " ", s) return s @@ -164,12 +187,14 @@ def _normalize_ddl(raw: str) -> str: def _compare_raw_ddl(local_ddl: str, remote_ddl: str) -> list[FieldDiff]: """Fallback: normalized string comparison for unparseable objects.""" if _normalize_ddl(local_ddl) != _normalize_ddl(remote_ddl): - return [FieldDiff( - field_name="raw_ddl", - local_value=local_ddl[:200] if local_ddl else None, - remote_value=remote_ddl[:200] if remote_ddl else None, - message="DDL definition differs (raw comparison)", - )] + return [ + FieldDiff( + field_name="raw_ddl", + local_value=local_ddl[:200] if local_ddl else None, + remote_value=remote_ddl[:200] if remote_ddl else None, + message="DDL definition differs (raw comparison)", + ) + ] return [] @@ -200,18 +225,30 @@ def compare_schemas(local: Schema, live: Schema) -> list[SchemaDiff]: live_obj = live_objs.get(name) if local_obj and not live_obj: - results.append(SchemaDiff( - name=name, obj_type=obj_type, status=DiffStatus.LOCAL_ONLY, - )) + results.append( + SchemaDiff( + name=name, + obj_type=obj_type, + status=DiffStatus.LOCAL_ONLY, + ) + ) elif live_obj and not local_obj: - results.append(SchemaDiff( - name=name, obj_type=obj_type, status=DiffStatus.REMOTE_ONLY, - )) + results.append( + SchemaDiff( + name=name, + obj_type=obj_type, + status=DiffStatus.REMOTE_ONLY, + ) + ) else: # Both exist — compare field_diffs: list[FieldDiff] = [] - if obj_type == "table" and isinstance(local_obj, TableDefinition) and isinstance(live_obj, TableDefinition): + if ( + obj_type == "table" + and isinstance(local_obj, TableDefinition) + and isinstance(live_obj, TableDefinition) + ): field_diffs = _compare_tables(local_obj, live_obj) else: # Fallback to raw DDL comparison for views, MVs, dicts @@ -220,13 +257,21 @@ def compare_schemas(local: Schema, live: Schema) -> list[SchemaDiff]: field_diffs = _compare_raw_ddl(local_ddl, live_ddl) if field_diffs: - results.append(SchemaDiff( - name=name, obj_type=obj_type, - status=DiffStatus.MODIFIED, field_diffs=field_diffs, - )) + results.append( + SchemaDiff( + name=name, + obj_type=obj_type, + status=DiffStatus.MODIFIED, + field_diffs=field_diffs, + ) + ) else: - results.append(SchemaDiff( - name=name, obj_type=obj_type, status=DiffStatus.IN_SYNC, - )) + results.append( + SchemaDiff( + name=name, + obj_type=obj_type, + status=DiffStatus.IN_SYNC, + ) + ) return results diff --git a/ch_migrate/display.py b/ch_migrate/display.py index a15d1fd..3573282 100644 --- a/ch_migrate/display.py +++ b/ch_migrate/display.py @@ -289,7 +289,7 @@ def render_snapshot_progress( total += count status_text = Text() - status_text.append("\nSnapshot saved to ", style="dim") + status_text.append(f"\nSnapshot saved to ", style="dim") status_text.append(output_dir, style="bold") status_text.append(f"\n{_count(total, 'object')} captured", style="green") if excluded: diff --git a/ch_migrate/gate.py b/ch_migrate/gate.py index 1492140..d28d9fe 100644 --- a/ch_migrate/gate.py +++ b/ch_migrate/gate.py @@ -4,13 +4,28 @@ from dotenv import load_dotenv +from ch_migrate import ui from ch_migrate.config import get_env_config, load_config from ch_migrate.connection import get_migration_state -from ch_migrate.lint import LintConfig, LintReport, Severity, lint_migrations +from ch_migrate.display import lint_finding_line +from ch_migrate.lint import GATE_RULES, LintConfig, LintReport, Severity, lint_migrations from ch_migrate.rebase import build_revision_graph from ch_migrate.statements import pending_revisions +def enforce_up_gate(project_root: Path, environment: str, skip_mv_check: bool = False) -> None: + """Print the pending upgrades' findings, and refuse the run if a gate rule failed.""" + try: + report = lint_pending_up(project_root, environment, skip_mv_check) + except Exception as error: + raise ui.Failure(f"Migration preflight failed: {error}") from error + if print_gate_findings(report): + raise ui.Failure( + "Migration gate failed; nothing was applied.", + "Fix SQL or add a reasoned in-file waiver.", + ) + + def lint_pending_up( project_root: Path, environment: str, skip_mv_check: bool = False ) -> LintReport: @@ -24,3 +39,21 @@ def lint_pending_up( env_config = get_env_config(environment, config_path) pending = pending_revisions(graph, get_migration_state(env_config).heads) return lint_migrations(versions, config=config, revisions=pending) + + +def print_gate_findings(report: LintReport) -> bool: + blocked = False + for finding in report.results: + gate_error = finding.rule in GATE_RULES and finding.severity == Severity.ERROR + blocked |= gate_error + severity = finding.severity.value + if severity == "error" and not gate_error: + severity = "warn" + position = finding.file or "config.yaml" + if finding.line is not None: + position += f":{finding.line}" + console = ui.err if gate_error else ui.out + console.print(lint_finding_line(finding, location=position, severity=severity)) + if finding.statement: + ui.detail(finding.statement, stderr=gate_error) + return blocked diff --git a/ch_migrate/introspect.py b/ch_migrate/introspect.py index 48a9c2b..d185d79 100644 --- a/ch_migrate/introspect.py +++ b/ch_migrate/introspect.py @@ -172,9 +172,7 @@ def affected_by_drop(self, name: str) -> list[ObjectNode]: _RE_SETTINGS = re.compile(r"SETTINGS\s+(.+)$", re.IGNORECASE | re.MULTILINE) _RE_CREATE_VIEW = re.compile( - r"CREATE\s+VIEW\s+(?:IF\s+NOT\s+EXISTS\s+)?" - r"(?:`?(\w+)`?\.)?`?(\w+)`?" - r"\s+AS\s+", + r"CREATE\s+VIEW\s+(?:IF\s+NOT\s+EXISTS\s+)?" r"(?:`?(\w+)`?\.)?`?(\w+)`?" r"\s+AS\s+", re.IGNORECASE, ) @@ -266,9 +264,7 @@ def _parse_single_column(line: str) -> ColumnDefinition | None: # Extract DEFAULT/MATERIALIZED/ALIAS default_kind = None default_expr = None - default_match = re.search( - r"\b(DEFAULT|MATERIALIZED|ALIAS)\s+(.+)$", rest, re.IGNORECASE - ) + default_match = re.search(r"\b(DEFAULT|MATERIALIZED|ALIAS)\s+(.+)$", rest, re.IGNORECASE) if default_match: default_kind = default_match.group(1).upper() default_expr = default_match.group(2).strip() @@ -481,15 +477,11 @@ def parse_create_dictionary(ddl: str) -> DictDefinition | None: source_type = source_match.group(1).lower() # Extract table from SOURCE(CLICKHOUSE(TABLE '...' DB '...')) - tbl_match = re.search( - r"TABLE\s+'([^']+)'", ddl[source_match.start() :], re.IGNORECASE - ) + tbl_match = re.search(r"TABLE\s+'([^']+)'", ddl[source_match.start() :], re.IGNORECASE) if tbl_match: source_table = tbl_match.group(1) - db_match = re.search( - r"DB\s+'([^']+)'", ddl[source_match.start() :], re.IGNORECASE - ) + db_match = re.search(r"DB\s+'([^']+)'", ddl[source_match.start() :], re.IGNORECASE) if db_match: source_db = db_match.group(1) @@ -548,9 +540,7 @@ def parse_create_statement( # --------------------------------------------------------------------------- -def list_objects( - client: Any, database: str, obj_type: str = "table" -) -> list[str]: +def list_objects(client: Any, database: str, obj_type: str = "table") -> list[str]: """List objects of a given type in a database. Args: @@ -575,16 +565,13 @@ def list_objects( }.get(obj_type, "1=1") result = client.query( - f"SELECT name FROM system.tables " - f"WHERE database = {{db:String}} AND {engine_filter}", + f"SELECT name FROM system.tables " f"WHERE database = {{db:String}} AND {engine_filter}", parameters={"db": database}, ) return [row[0] for row in result.result_rows] -def get_create_statement( - client: Any, database: str, name: str, obj_type: str = "table" -) -> str: +def get_create_statement(client: Any, database: str, name: str, obj_type: str = "table") -> str: """Get the CREATE statement for an object. Args: diff --git a/ch_migrate/json_output.py b/ch_migrate/json_output.py deleted file mode 100644 index 801c7db..0000000 --- a/ch_migrate/json_output.py +++ /dev/null @@ -1,152 +0,0 @@ -"""Versioned CLI documents; stdout contains only JSON in machine-readable mode.""" - -from __future__ import annotations - -import json -import sys -from contextlib import redirect_stdout -from typing import Any, NoReturn - -import click - -from ch_migrate.diff import DiffStatus, SchemaDiff -from ch_migrate.downgrade import irreversible_reason -from ch_migrate.lint import GATE_RULES, LintReport, Severity -from ch_migrate.rebase import RevisionGraph - - -class JsonCommand(click.Command): - """Keep diagnostics off stdout and return a document for command errors.""" - - def parse_args(self, ctx: click.Context, args: list[str]) -> list[str]: - json_requested = "--json" in args - try: - return super().parse_args(ctx, args) - except click.ClickException as error: - if not json_requested: - raise - emit_json(self.name or "", {"error": error.format_message()}) - raise click.exceptions.Exit(2) from error - - def invoke(self, ctx: click.Context) -> Any: - if not ctx.params.get("json_output"): - return super().invoke(ctx) - ctx.meta["json_stdout"] = sys.stdout - with redirect_stdout(sys.stderr): - try: - return super().invoke(ctx) - except click.exceptions.Exit: - raise - except Exception as error: - message = ( - error.format_message() - if isinstance(error, click.ClickException) - else str(error) - ) - emit_json(self.name or "", {"error": message}) - raise click.exceptions.Exit(_error_exit(self.name)) from error - - -def emit_json(command: str, payload: dict[str, Any]) -> None: - """Write one schema-versioned document to the caller's original stdout.""" - ctx = click.get_current_context(silent=True) - stream = ctx.meta.get("json_stdout") if ctx else None - document = {"schema_version": 1, "command": command, **payload} - click.echo(json.dumps(document, ensure_ascii=False, allow_nan=False), file=stream) - - -def command_failure(message: str) -> NoReturn: - """Preserve human failure behavior while giving JSON callers a real error.""" - ctx = click.get_current_context() - if ctx.params.get("json_output"): - emit_json(ctx.command.name or "", {"error": message}) - raise click.exceptions.Exit(_error_exit(ctx.command.name)) - click.echo(message, err=True) - raise click.exceptions.Exit(1) - - -def status_document(graph: RevisionGraph, heads: set[str], database: str) -> dict[str, Any]: - applied = set() - for head in heads: - if head in graph.migrations: - applied.update(graph.walk_to_root(head)) - applied.intersection_update(graph.migrations) - script_heads = set(graph.heads()) - return { - "database": database, - "current_heads": sorted(heads), - "script_heads": sorted(script_heads), - "pending": sorted(set(graph.migrations) - applied), - "applied": sorted(applied), - "at_head": heads == script_heads, - } - - -def history_document(graph: RevisionGraph, applied: set[str] | None) -> dict[str, Any]: - return { - "revisions": [ - { - "revision": migration.revision, - "down_revisions": list(migration.down_revisions), - "description": migration.description, - "create_date": migration.create_date, - "path": str(migration.path), - "applied": migration.revision in applied if applied is not None else None, - "irreversible": irreversible_reason(graph, migration.revision), - } - for migration in graph.migrations.values() - ] - } - - -def lint_document(report: LintReport) -> dict[str, Any]: - findings = [ - { - "rule": result.rule, - "severity": result.severity.value, - "blocking": result.rule in GATE_RULES and result.severity == Severity.ERROR, - "file": result.file, - "line": result.line, - "message": result.message, - "waived": result.waived, - } - for result in report.results - ] - return { - "findings": findings, - "counts": { - "total": len(findings), - "error": report.error_count, - "warning": report.warning_count, - "info": report.info_count, - "blocking": sum(finding["blocking"] for finding in findings), - "waived": sum(finding["waived"] is not None for finding in findings), - }, - } - - -def diff_document(diffs: list[SchemaDiff]) -> dict[str, Any]: - return { - "objects": [ - { - "type": item.obj_type, - "name": item.name, - "status": item.status.value, - "details": [ - { - "field": detail.field_name, - "local": detail.local_value, - "remote": detail.remote_value, - "message": detail.message, - } - for detail in item.field_diffs - ], - } - for item in diffs - ], - "in_sync": all(item.status == DiffStatus.IN_SYNC for item in diffs), - } - - -def _error_exit(command: str | None) -> int: - return 1 if command == "lint" else 2 diff --git a/ch_migrate/lint.py b/ch_migrate/lint.py index 320376d..a0efddb 100644 --- a/ch_migrate/lint.py +++ b/ch_migrate/lint.py @@ -9,7 +9,6 @@ from pathlib import Path from typing import Any -from ch_migrate import ui from ch_migrate.alembic_env import has_current_env from ch_migrate.baseline import baseline_exemptions, normalize_baseline from ch_migrate.idempotency import classify_idempotency, waiver_reason @@ -40,7 +39,6 @@ class LintResult: file: str | None = None line: int | None = None statement: str | None = None - waived: str | None = None @dataclass @@ -68,6 +66,7 @@ def info_count(self) -> int: class LintConfig: """Lint configuration loaded from config.yaml.""" + large_table_threshold: int = 100_000_000 rules: dict[str, Severity] = field(default_factory=dict) mv_validation_cutoff: str | None = None gate_baseline: tuple[str, ...] = () @@ -78,15 +77,10 @@ def from_config(cls, config: dict[str, Any]) -> LintConfig: if not lint_section: return cls() + threshold = lint_section.get("large_table_threshold", 100_000_000) rules_raw = lint_section.get("rules", {}) - if "large_table_threshold" in lint_section or "large_table_mutation" in rules_raw: - ui.warn( - "Deprecated: large_table_mutation/large_table_threshold are ignored; use ch-migrate plan." - ) rules = {} for name, level in rules_raw.items(): - if name == "large_table_mutation": - continue if name in GATE_RULES and level != Severity.ERROR: raise ValueError( f"lint.rules.{name} must remain error; use an in-file waiver " @@ -100,6 +94,7 @@ def from_config(cls, config: dict[str, Any]) -> LintConfig: cutoff = lint_section.get("mv_validation_cutoff") return cls( + large_table_threshold=threshold, rules=rules, mv_validation_cutoff=cutoff, gate_baseline=normalize_baseline(lint_section.get("gate_baseline")), @@ -382,6 +377,12 @@ def check(self, sql: str, **kwargs: Any) -> list[LintResult]: elif reason is not None: severity = Severity.INFO message = f"Idempotency waiver: {reason}" + elif not kwargs.get("readable", True): + message = ( + "This SQL only exists at run time, so ch-migrate can't check that running it " + "twice is safe. Pass it as a string literal, or put " + "# ch-migrate: allow-non-idempotent on the line above" + ) else: check = classify_idempotency(sql) if check.status == "ok": @@ -391,17 +392,7 @@ def check(self, sql: str, **kwargs: Any) -> list[LintResult]: "ch-migrate: allow-non-idempotent waiver" ) first_line = sql.strip().splitlines()[0] if sql.strip() else "" - return [ - LintResult( - self.name, - message, - severity, - kwargs.get("file_path"), - 1, - first_line, - waived=reason or None, - ) - ] + return [LintResult(self.name, message, severity, kwargs.get("file_path"), 1, first_line)] class StandaloneSetRule(LintRule): @@ -508,6 +499,66 @@ def check(self, sql: str, **kwargs: Any) -> list[LintResult]: # --------------------------------------------------------------------------- +class LargeTableMutationRule(LintRule): + """Flags ALTER on tables above a configurable row threshold.""" + + name = "large_table_mutation" + default_severity = Severity.WARN + requires_db = True + + _RE_ALTER_TABLE = re.compile( + r"\bALTER\s+TABLE\s+(?:`?(\w+|\{[^}]+\})`?\.)?`?(\w+)`?", + re.IGNORECASE, + ) + + def check(self, sql: str, **kwargs: Any) -> list[LintResult]: + config = kwargs.get("config") or LintConfig() + severity = self.get_severity(config) + if severity == Severity.OFF: + return [] + + client = kwargs.get("client") + database = kwargs.get("database") + if not client or not database: + return [] + + results: list[LintResult] = [] + file_path = kwargs.get("file_path") + threshold = config.large_table_threshold + + for match in self._RE_ALTER_TABLE.finditer(sql): + db = match.group(1) + if db is None or db.startswith("{"): + db = database + table_name = match.group(2) + try: + result = client.query( + "SELECT count() FROM system.parts " + "WHERE database = {db:String} AND table = {tbl:String} AND active", + parameters={"db": db, "tbl": table_name}, + ) + if result.result_rows: + row_count = result.result_rows[0][0] + if row_count > threshold: + line = sql[: match.start()].count("\n") + 1 + results.append( + LintResult( + rule=self.name, + message=( + f"ALTER on '{table_name}' which has {row_count:,} parts " + f"(threshold: {threshold:,})" + ), + severity=severity, + file=file_path, + line=line, + ) + ) + except Exception: + pass + + return results + + class MVDependencyRule(LintRule): """Flags operations on tables that have materialized view dependencies.""" @@ -627,6 +678,7 @@ def check(self, sql: str, **kwargs: Any) -> list[LintResult]: ] RUNTIME_RULES: list[LintRule] = [ + LargeTableMutationRule(), MVDependencyRule(), ] @@ -705,8 +757,9 @@ def _lint_statement( for rule in rules: if isinstance(rule, MVDeclarationRule) or rule.get_severity(scope.config) == Severity.OFF: continue - if statement.rebuild is not None and rule.name in GATE_RULES: - continue # The operation guards its internal steps with durable rebuild state. + # SQL that only exists at run time can't be parsed; only its waiver is checked. + if not statement.readable and not isinstance(rule, IdempotencyRule): + continue findings = rule.check( statement.sql, file_path=statement.source, @@ -716,6 +769,7 @@ def _lint_statement( graph=scope.graph, comments=statement.comments, session_safe=scope.session_safe, + readable=statement.readable, ) for finding in findings: finding.file = statement.source diff --git a/ch_migrate/migration_runner.py b/ch_migrate/migration_runner.py index f1d9ee9..f444fc6 100644 --- a/ch_migrate/migration_runner.py +++ b/ch_migrate/migration_runner.py @@ -1,27 +1,250 @@ -"""Own the upgrade process so SIGKILL cannot leave a child advancing revisions.""" +"""Own the upgrade process so SIGKILL cannot leave a child advancing revisions. + +`ch-migrate up` runs Alembic in this process and reports in ch-migrate's own words, +as runner.py does for `down`: one line per migration, setup chatter dropped, and a +failure reduced to the migration, what went wrong and what to do next. Waiting +progress is drawn by a display the waiter receives through the Alembic config. +""" from __future__ import annotations +import logging import os +import re +import traceback +from contextlib import contextmanager +from dataclasses import dataclass from pathlib import Path from alembic import command from alembic.config import Config +from ch_migrate import ui +from ch_migrate.runner import ( + _RUNNING, + _SELF_DESCRIBING, + _SETUP_CHATTER, + _Progress, + _report_success, + _start_migration, +) +from ch_migrate.sql import clean_database_error +from ch_migrate.waiting.display import WaitDisplay +from ch_migrate.waiting.errors import ( + FailedMutation, + MutationFailed, + WaitingError, + WaitTimeout, +) +from ch_migrate.waiting.reruns import InterruptedStep -def run_upgrade(environment: str, revision: str, timeout: float | None) -> None: - """Run Alembic directly: progress streams live and the CLI owns the waiter.""" - config = Config(str(Path.cwd() / "alembic.ini")) - config.attributes["ch_migrate_environment"] = environment - config.attributes["ch_migrate_command"] = "upgrade" - config.attributes["ch_migrate_timeout"] = timeout +_LOCATION = re.compile( + r"^(?Pmigrations/sql/\S+) \(statement \d+ of \d+, line (?P\d+)\): " +) + + +@dataclass(frozen=True) +class Upgrade: + """One `ch-migrate up`: where, how far, how long to wait, what to rerun, what to show.""" + + environment: str + revision: str = "head" + timeout: float | None = None + verbose: bool = False + rerun_interrupted: tuple[str, ...] = () + + +def run_upgrade(upgrade: Upgrade) -> int: + """Apply migrations in this process, one reported line each; return the exit status.""" + progress, display = _Progress(), WaitDisplay() + config = _alembic_config(upgrade, progress, display) # Project helpers intentionally read these during migration execution only. saved = {name: os.environ.get(name) for name in ("CH_DATABASE", "CH_CLUSTER")} try: - command.upgrade(config, revision) + with _restored_loggers("alembic", "ch_migrate"): + command.upgrade(config, upgrade.revision) + except KeyboardInterrupt: + display.stop() + _report_interrupt(upgrade.environment, progress) + return 130 + except Exception as error: + display.stop() + _report_failure(upgrade, progress, error) + return 1 + finally: + _restore_environment(saved) + _report_success(upgrade.environment, upgrade.revision, progress) + return 0 + + +class _MigrationLog(logging.Handler): + """Turns Alembic's log records into ch-migrate's lines, as runner.py does for its output.""" + + def __init__(self, progress: _Progress, display: WaitDisplay) -> None: + super().__init__(logging.INFO) + self.progress, self.display = progress, display + + def emit(self, record: logging.LogRecord) -> None: + message = record.getMessage() + running = _RUNNING.match(message) + if running: + _start_migration(running, self.progress) + self.display.start_migration(self.progress.current) + elif message.startswith(_SETUP_CHATTER): + return + elif record.levelno >= logging.WARNING: + ui.warn(message) + else: + ui.detail(message) + + +def _alembic_config(upgrade: Upgrade, progress: _Progress, display: WaitDisplay) -> Config: + """The project's Alembic config, carrying what env.py needs from this invocation.""" + config = Config(str(Path.cwd() / "alembic.ini")) + config.attributes.update( + ch_migrate_environment=upgrade.environment, + ch_migrate_command="upgrade", + # `ch-migrate up` has already run the gate; env.py runs it only for a direct upgrade. + ch_migrate_gate_checked=True, + ch_migrate_rerun_interrupted=upgrade.rerun_interrupted, + ch_migrate_timeout=upgrade.timeout, + ch_migrate_display=display, + ch_migrate_log_handler=_MigrationLog(progress, display), + ) + return config + + +def _report_interrupt(environment: str, progress: _Progress) -> None: + ui.error(f"Interrupted{' during ' + progress.current if progress.current else ''}") + ui.hint( + "Work ClickHouse already started keeps running, and the migration is not recorded as " + f"applied. Run `ch-migrate up {environment}` again: it waits for that work to finish, " + "then runs the migration again, or stops first if a waived statement may already have " + "applied.", + stderr=True, + ) + + +def _report_failure(upgrade: Upgrade, progress: _Progress, error: Exception) -> None: + where = progress.current or "Upgrade" + waiting = _waiting_error(error) + if waiting is not None and (location := _LOCATION.match(str(error))): + where = f"{where} ({location['path']}:{location['line']})" + if isinstance(waiting, MutationFailed): + _report_mutation_failure(upgrade.environment, where, waiting.failure) + elif isinstance(waiting, WaitTimeout): + _report_timeout(upgrade.environment, where, waiting) + elif isinstance(waiting, InterruptedStep): + _report_interrupted_step(upgrade.environment, waiting) + elif waiting is not None: + ui.error(f"{where}: {waiting}") + else: + _report_statement_failure(upgrade, where, error) + + +def _report_mutation_failure(environment: str, where: str, failure: FailedMutation) -> None: + on_host = "" if failure.host == "shared" else f" on {failure.host}" + if failure.foreign: + ui.error( + f"{where} is blocked: mutation {failure.mutation} on {failure.table}, started " + f"before this run, failed{on_host}" + ) + else: + ui.error(f"{where} failed: mutation {failure.mutation} on {failure.table} failed{on_host}") + ui.detail(clean_database_error(failure.reason), stderr=True) + ui.hint( + "It was not recorded as applied. ClickHouse keeps retrying a failed mutation, and later " + f"mutations on {failure.table} wait behind it.", + stderr=True, + ) + ui.hint( + "If the cause is temporary, such as memory or disk, fix it and run " + f"`ch-migrate up {environment}` to wait for the retry.", + stderr=True, + ) + ui.hint( + f"If the statement itself is wrong, an operator stops it{on_host} with {failure.kill}, " + f"then fixes the SQL and runs `ch-migrate up {environment}` again.", + stderr=True, + ) + + +def _report_timeout(environment: str, where: str, timeout: WaitTimeout) -> None: + ui.error(f"Timed out after {timeout.elapsed:.1f}s waiting on {where}") + ui.detail(f"Pending: {timeout.pending}", stderr=True) + ui.hint( + "ClickHouse keeps doing the work, and the migration is not recorded as applied. " + f"Run `ch-migrate up {environment}` again: it waits for that work to finish, then runs " + "the migration again, or stops first if a waived statement may already have applied.", + stderr=True, + ) + + +def _report_interrupted_step(environment: str, stop: InterruptedStep) -> None: + revision = stop.step.revision[:8] + ui.error( + f"{revision} was interrupted on an earlier run, and ch-migrate can't vouch for " + "running it again" + ) + for item in stop.unvouched: + where = f"{item.statement.source}:{item.statement.line}" + ui.detail(f"{where} {item.why}: {item.statement.sql.splitlines()[0]}", stderr=True) + ui.hint( + "Some of its statements may already have applied. Check what the earlier run changed, " + f"then run `ch-migrate up {environment} --rerun-interrupted {revision}`.", + stderr=True, + ) + + +def _report_statement_failure(upgrade: Upgrade, where: str, error: Exception) -> None: + ui.error(f"{where} failed") + ui.detail(_reason(error), stderr=True) + ui.hint( + "It was not recorded as applied. Anything it ran before the failure stays in place, " + f"so make the SQL safe to re-run, then run `ch-migrate up {upgrade.environment}` again.", + stderr=True, + ) + if upgrade.verbose: + traceback.print_exception(error) + else: + ui.hint("Add `--verbose` to see the full traceback.", stderr=True) + + +def _reason(error: Exception) -> str: + lines = str(error).strip().splitlines() + message = clean_database_error(lines[0]) if lines else "" + kind = type(error).__name__ + if not message: + return kind + return message if kind in _SELF_DESCRIBING else f"{kind}: {message}" + + +def _waiting_error(error: BaseException) -> WaitingError | None: + current: BaseException | None = error + seen: set[int] = set() + while current is not None and id(current) not in seen: + if isinstance(current, WaitingError): + return current + seen.add(id(current)) + current = getattr(current, "orig", None) or current.__cause__ or current.__context__ + return None + + +@contextmanager +def _restored_loggers(*names: str): + saved = [(logging.getLogger(n), logging.getLogger(n).handlers[:]) for n in names] + states = [(logger.propagate, logger.level) for logger, _ in saved] + try: + yield finally: - for name, value in saved.items(): - if value is None: - os.environ.pop(name, None) - else: - os.environ[name] = value + for (logger, handlers), (propagate, level) in zip(saved, states): + logger.handlers, logger.propagate = handlers, propagate + logger.setLevel(level) + + +def _restore_environment(saved: dict[str, str | None]) -> None: + for name, value in saved.items(): + if value is None: + os.environ.pop(name, None) + else: + os.environ[name] = value diff --git a/ch_migrate/mv_validate.py b/ch_migrate/mv_validate.py index 0ba25bf..50aaa6d 100644 --- a/ch_migrate/mv_validate.py +++ b/ch_migrate/mv_validate.py @@ -121,9 +121,7 @@ def _extract_create_date(path: Path) -> str | None: # Handles: {db}.table, db.table, table (no db prefix) # Handles backtick-quoted identifiers and comma-separated user lists _RE_GRANT_INSERT = re.compile( - r"GRANT\s+INSERT\s+ON\s+" - r"(?:(?:\{[^}]*\}|`?\w+`?)\.)?`?(\w+)`?" - r"\s+TO\s+(.+?)(?:;|\n|\Z)", + r"GRANT\s+INSERT\s+ON\s+" r"(?:(?:\{[^}]*\}|`?\w+`?)\.)?`?(\w+)`?" r"\s+TO\s+(.+?)(?:;|\n|\Z)", re.IGNORECASE, ) @@ -162,11 +160,7 @@ def _parse_user_list(users_str: str) -> list[str]: Handles backtick-quoted identifiers and strips whitespace. """ - return [ - u.strip().strip("`").lower() - for u in users_str.split(",") - if u.strip().strip("`") - ] + return [u.strip().strip("`").lower() for u in users_str.split(",") if u.strip().strip("`")] def _find_grant_inserts(content: str) -> list[tuple[str, str]]: @@ -203,8 +197,7 @@ def _find_row_policies(content: str) -> list[tuple[str, str]]: Returns list of (policy_name, table_name) tuples, lowercased. """ return [ - (m.group(1).lower(), m.group(2).lower()) - for m in _RE_CREATE_ROW_POLICY.finditer(content) + (m.group(1).lower(), m.group(2).lower()) for m in _RE_CREATE_ROW_POLICY.finditer(content) ] @@ -236,9 +229,7 @@ def _read_migration_sql_from_content(content: str, path: Path) -> str: """ sql_parts: list[str] = [] - for match in re.finditer( - r'op\.execute\(\s*(?:f?"""(.*?)"""|f?"([^"]*)")', content, re.DOTALL - ): + for match in re.finditer(r'op\.execute\(\s*(?:f?"""(.*?)"""|f?"([^"]*)")', content, re.DOTALL): sql_parts.append(match.group(1) or match.group(2) or "") for match in re.finditer(r'(?:read_sql|run_sql)\(\s*["\']([^"\']+)["\']', content): @@ -308,18 +299,13 @@ def _check_grant_inserts( target = decl.target_table.lower() for user in decl.inserting_users: - has_grant = any( - table == target and u == user.lower() for table, u in grant_inserts - ) + has_grant = any(table == target and u == user.lower() for table, u in grant_inserts) if not has_grant: errors.append( MVValidationError( file=file_name, mv_name=decl.mv_name, - message=( - f"Missing: GRANT INSERT ON {{db}}.{decl.target_table} " - f"TO {user}" - ), + message=(f"Missing: GRANT INSERT ON {{db}}.{decl.target_table} " f"TO {user}"), ) ) @@ -336,18 +322,13 @@ def _check_grant_selects( target = decl.target_table.lower() for user in decl.select_users: - has_grant = any( - table == target and u == user.lower() for table, u in grant_selects - ) + has_grant = any(table == target and u == user.lower() for table, u in grant_selects) if not has_grant: errors.append( MVValidationError( file=file_name, mv_name=decl.mv_name, - message=( - f"Missing: GRANT SELECT ON {{db}}.{decl.target_table} " - f"TO {user}" - ), + message=(f"Missing: GRANT SELECT ON {{db}}.{decl.target_table} " f"TO {user}"), ) ) @@ -365,8 +346,7 @@ def _check_source_rls( for user in decl.source_rls_users: has_policy = any( - table == source - and user.lower() in [u.strip() for u in users_str.split(",")] + table == source and user.lower() in [u.strip() for u in users_str.split(",")] for table, users_str in permissive_policies ) if not has_policy: @@ -395,8 +375,7 @@ def _check_target_rls( for policy_name in decl.target_rls_policies: has_policy = any( - table == target and policy_name.lower() in policy - for policy, table in row_policies + table == target and policy_name.lower() in policy for policy, table in row_policies ) if not has_policy: errors.append( @@ -531,24 +510,16 @@ def validate_mv_migrations( continue # Core check: GRANT INSERT on target for each inserting user - errors.extend( - _check_grant_inserts(decl, grant_inserts, file_name) - ) + errors.extend(_check_grant_inserts(decl, grant_inserts, file_name)) # Optional checks (only if the declaration includes them) if decl.select_users: - errors.extend( - _check_grant_selects(decl, grant_selects, file_name) - ) + errors.extend(_check_grant_selects(decl, grant_selects, file_name)) if decl.source_rls_users: - errors.extend( - _check_source_rls(decl, permissive_policies, file_name) - ) + errors.extend(_check_source_rls(decl, permissive_policies, file_name)) if decl.target_rls_policies: - errors.extend( - _check_target_rls(decl, row_policies, file_name) - ) + errors.extend(_check_target_rls(decl, row_policies, file_name)) return errors diff --git a/ch_migrate/plan.py b/ch_migrate/plan.py deleted file mode 100644 index 141d88c..0000000 --- a/ch_migrate/plan.py +++ /dev/null @@ -1,453 +0,0 @@ -"""Read-only planning of pending upgrades against the current live schema.""" - -from __future__ import annotations - -import shlex -from dataclasses import asdict, dataclass, replace -from graphlib import TopologicalSorter -from pathlib import Path -from typing import Any - -from clickhouse_connect.driver.binding import quote_identifier -from rich.console import Console - -from ch_migrate.classify import ( - Classification, - _alter_actions, - _identifier, - _name, - _order_expressions, - _split_settings, - _table_definition, - _tokens, - classify, -) -from ch_migrate.config import get_env_config, load_config -from ch_migrate.connection import get_client -from ch_migrate.introspect import DependencyGraph, Schema, get_dependencies, get_live_schema -from ch_migrate.json_output import lint_document -from ch_migrate.lint import LintConfig, lint_migrations -from ch_migrate.rebase import build_revision_graph -from ch_migrate.rebuild_ddl import build_definition -from ch_migrate.rebuild_preflight import RebuildRequest, inspect_rebuild -from ch_migrate.rebuild_types import RebuildOptions -from ch_migrate.statements import ( - MigrationStatement, - migration_statements, - pending_revisions, -) -from ch_migrate.version_table import ( - VersionTableState, - assert_version_mutations_healthy, - inspect_version_table, -) - - -@dataclass(frozen=True) -class RewriteSize: - precision: str - compressed_bytes: int - uncompressed_bytes: int - part_count: int - columns: list[str] - partition_id: str | None - - -def build_plan(root: Path, environment: str) -> dict[str, Any]: - """Resolve pending revisions without importing them or issuing a read barrier.""" - versions = root / "migrations" / "versions" - if not versions.is_dir(): - raise ValueError("migrations/versions/ not found") - config_path = root / "config.yaml" - lint_config = LintConfig.from_config(load_config(config_path)) - env = get_env_config(environment, config_path) - client = get_client(env) - try: - return _build_document(client, _PlanInput(root, environment, env, lint_config)) - finally: - client.close() - - -def render_plan(document: dict[str, Any]) -> None: - """Render the same facts used by JSON; no independent inspection or estimates.""" - console = Console(markup=False, highlight=False, soft_wrap=True) - console.print(f"Plan: {document['environment']} / {document['database']}") - for warning in document["warnings"]: - console.print(f"Warning: {warning}") - for migration in document["migrations"]: - console.print(f"\nRevision {migration['revision']} ({migration['path']})", style="bold") - for statement in migration["statements"]: - _render_statement(console, statement) - console.print("\nLint findings:") - for finding in document["findings"]: - _render_finding(console, finding) - console.print( - "Counts: " + ", ".join(f"{key}={value}" for key, value in document["counts"].items()) - ) - console.print("Gate would refuse up: " + ("yes" if document["gate_would_refuse"] else "no")) - - -@dataclass(frozen=True) -class _PlanInput: - root: Path - environment: str - env: dict[str, Any] - lint_config: LintConfig - - -@dataclass(frozen=True) -class _Scope: - client: Any - schema: Schema - dependencies: DependencyGraph - env: dict[str, Any] - findings: list[dict[str, Any]] - environment: str - deployment: VersionTableState - revision: str = "" - - -@dataclass(frozen=True) -class _Rewrite: - database: str - table: str - columns: tuple[str, ...] - partition_id: str | None - - -def _build_document(client: Any, request: _PlanInput) -> dict[str, Any]: - versions = request.root / "migrations" / "versions" - graph = build_revision_graph(versions) - heads, warnings, deployment = _read_heads(client, request.env) - pending = pending_revisions(graph, heads) - report = lint_migrations( - versions, - config=request.lint_config, - client=client, - database=request.env["database"], - revisions=pending, - ) - lint = lint_document(report) - database = request.env["database"] - scope = _Scope( - client, - get_live_schema(client, database), - get_dependencies(client, database), - request.env, - lint["findings"], - request.environment, - deployment, - ) - ordered = TopologicalSorter( - {rev: item.down_revisions for rev, item in graph.migrations.items()} - ) - migrations = [] - for revision in ordered.static_order(): - if revision not in graph.migrations: - raise ValueError(f"Missing parent revision {revision}") - if revision not in pending: - continue - migration = graph.migrations[revision] - revision_scope = replace(scope, revision=revision) - statements = [ - _plan_statement(statement, revision_scope) - for statement in migration_statements(migration.path) - if statement.direction == "upgrade" - ] - migrations.append( - { - "revision": revision, - "path": str(migration.path.relative_to(request.root)), - "statements": statements, - } - ) - return { - "database": database, - "environment": request.environment, - "warnings": warnings, - "gate_would_refuse": bool(lint["counts"]["blocking"]), - **lint, - "migrations": migrations, - } - - -def _read_heads(client: Any, env: dict[str, Any]) -> tuple[set[str], list[str], VersionTableState]: - state = inspect_version_table(client, env["database"], env.get("cluster")) - assert_version_mutations_healthy(client, state) - warnings = [state.warning()] if state.warning() else [] - if not state.table_engine: - return set(), warnings, state - rows = client.query( - "SELECT count() FROM system.mutations WHERE database = {db:String} " - "AND table = 'alembic_version' AND is_done = 0", - parameters={"db": env["database"]}, - ).result_rows - if rows[0][0]: - raise RuntimeError("Version bookkeeping is unfinished; reconcile it before planning") - final = " FINAL" if state.table_engine.endswith("ReplacingMergeTree") else "" - heads = client.query( - f"SELECT version_num FROM {quote_identifier(env['database'])}.alembic_version{final}" - ) - if "Replicated" in state.table_engine or "Shared" in state.table_engine: - warnings.append( - "Read-only plan observes the connected replica; it does not synchronize replicas or freeze concurrent migrations." - ) - return {row[0] for row in heads.result_rows}, warnings, state - - -def _plan_statement(statement: MigrationStatement, scope: _Scope) -> dict[str, Any]: - sql = statement.sql.replace("{db}", scope.env["database"]) - if scope.env.get("cluster"): - sql = sql.replace("{cluster}", scope.env["cluster"]) - rebuild_call = statement.rebuild - if rebuild_call: - rebuild_call = replace( - rebuild_call, table=rebuild_call.table.replace("{db}", scope.env["database"]) - ) - statement = replace(statement, sql=sql, rebuild=rebuild_call) - classification = classify(statement, scope.schema) - table = _table_definition(scope.schema, classification.table or "") - downstream = scope.dependencies.affected_by_drop(table.name) if table else [] - size = _rewrite_size(sql, classification, scope) - rebuild = None - if classification.kind == "rebuild": - if table is None: - raise ValueError(f"Cannot inspect rebuild source {classification.table!r}") - rebuild = asdict(_rebuild_assessment(statement, table, scope)) - findings = [ - item - for item in scope.findings - if item["file"] == statement.source and item["line"] == statement.line - ] - return { - "sql": sql, - "file": statement.source, - "line": statement.line, - "classification": asdict(classification), - "size": asdict(size) if size else None, - "downstream": [ - {"name": node.name, "type": node.obj_type} - for node in downstream - if node.obj_type in ("materialized_view", "dictionary") - ], - "findings": findings, - "rebuild": rebuild, - **( - {"suggested_command": _suggest_rebuild(scope.environment, table.name)} - if classification.kind == "rebuild" - else {} - ), - } - - -def _suggest_rebuild(environment: str, table: str) -> str: - return shlex.join( - ["ch-migrate", "new", environment, f"rebuild_{table}", "--table", table, "--rebuild"] - ) - - -def _rebuild_assessment(statement: MigrationStatement, table: Any, scope: _Scope): - call = statement.rebuild - if call: - options = RebuildOptions( - scope.env["database"], - table.name, - scope.revision, - 0, - scope.deployment.on_cluster, - call.select, - call.allow_unacknowledged_async_loss, - ) - target = build_definition(table, statement.sql, options).target - else: - target = _rebuild_target(table, statement.sql) - request = RebuildRequest( - scope.env["database"], - table, - target, - scope.deployment.replica_cluster, - bool(call and call.allow_unacknowledged_async_loss), - writers_on_this_service_only=bool(call and call.writers_on_this_service_only), - ) - return inspect_rebuild(scope.client, request) - - -def _rewrite_size(sql: str, classification: Classification, scope: _Scope) -> RewriteSize | None: - tokens = _tokens(sql) - if not classification.table or ( - classification.kind != "mutation" and tokens[0].upper() != "UPDATE" - ): - return None - database, _, table = classification.table.rpartition(".") - columns = _exact_columns(tokens, scope.schema) - definition = _table_definition(scope.schema, classification.table) - partition = _partition_id(tokens, definition) - request = _Rewrite(database or scope.env["database"], table, columns, partition) - return _measure_rewrite(scope.client, request) - - -def _measure_rewrite(client: Any, request: _Rewrite) -> RewriteSize: - parameters = {"db": request.database, "table": request.table} - predicate = "database = {db:String} AND table = {table:String} AND active = 1" - if request.partition_id is not None: - predicate += " AND partition_id = {partition:String}" - parameters["partition"] = request.partition_id - parts = client.query( - "SELECT sum(data_compressed_bytes), sum(data_uncompressed_bytes), count(), " - "countIf(part_type = 'Compact') FROM system.parts WHERE " + predicate, - parameters=parameters, - ).result_rows[0] - # Compact parts share one data file; per-column counters are zero, not exact - # rewrite bytes. Keep the whole-part ceiling rather than claiming zero work. - exact = bool(request.columns) and not parts[3] - row = parts - if exact: - parameters["columns"] = list(request.columns) - row = client.query( - "SELECT sum(column_data_compressed_bytes), sum(column_data_uncompressed_bytes), " - "uniqExact(name) FROM system.parts_columns WHERE " - + predicate - + " AND column IN {columns:Array(String)}", - parameters=parameters, - ).result_rows[0] - return RewriteSize( - "exact" if exact else "ceiling", - int(row[0]), - int(row[1]), - int(row[2]), - list(request.columns), - request.partition_id, - ) - - -def _alter_clauses(tokens: tuple[str, ...]) -> tuple[tuple[str, ...], ...]: - index = 4 if tuple(token.upper() for token in tokens[2:4]) == ("IF", "EXISTS") else 2 - _, index = _name(tokens, index) - if tuple(token.upper() for token in tokens[index : index + 2]) == ("ON", "CLUSTER"): - index += 3 - actions, _ = _split_settings(tokens[index:]) - return _alter_actions(actions) - - -def _exact_columns(tokens: tuple[str, ...], schema: Schema) -> tuple[str, ...]: - if tuple(token.upper() for token in tokens[:2]) != ("ALTER", "TABLE"): - return () - columns = [] - table, _ = _name(tokens, 4 if tokens[2:4] == ("IF", "EXISTS") else 2) - actions = _alter_clauses(tokens) - if len(actions) > 1 and any( - "PARTITION" in (token.upper() for token in action) for action in actions - ): - return () - for action in actions: - if classify(f"ALTER TABLE {table} " + " ".join(action), schema).kind == "metadata": - continue - words = tuple(token.upper() for token in action) - if words[:2] not in (("MODIFY", "COLUMN"), ("DROP", "COLUMN"), ("CLEAR", "COLUMN")): - return () - index = 4 if words[2:4] == ("IF", "EXISTS") else 2 - if len(action) <= index: - return () - columns.append(_identifier(action[index])) - return tuple(dict.fromkeys(columns)) - - -def _partition_id(tokens: tuple[str, ...], table: Any) -> str | None: - words = tuple(token.upper() for token in tokens) - explicit = [ - index - for index in range(len(words) - 3) - if words[index : index + 3] == ("IN", "PARTITION", "ID") - ] - if explicit: - values = {tokens[index + 3].strip("'") for index in explicit} - # Mixed ALTER actions may affect the whole table; never understate their ceiling. - actions = _alter_clauses(tokens) if words[:2] == ("ALTER", "TABLE") else (tokens,) - if len(values) == 1 and all( - "PARTITION" in tuple(token.upper() for token in action) for action in actions - ): - return values.pop() - if "WHERE" not in words: - return None - predicate, _ = _split_settings(tokens[words.index("WHERE") + 1 :]) - # Only an entire simple equality can narrow the ceiling. OR, casts and tuples remain table-wide. - if len(predicate) == 3 and predicate[1] == "=" and _identifier(predicate[0]) == "_partition_id": - if predicate[2].startswith("'") and predicate[2].endswith("'"): - return predicate[2][1:-1] - # Numeric identity partition keys have the same printable ID as their value. - if table and len(predicate) >= 3 and predicate[-2] == "=" and predicate[-1].isdigit(): - key_tokens = _tokens(table.partition_by or "") - if key_tokens[:2] in (("toYYYYMM", "("), ("toYYYYMMDD", "("), ("toYear", "(")): - if predicate[:-2] == key_tokens: - return predicate[-1] - key = table.partition_by or "" - column = next((col for col in table.columns if col.name == key), None) - if ( - len(predicate) == 3 - and column - and _identifier(predicate[0]) == key - and column.type.startswith("UInt") - ): - return predicate[-1] - return None - - -def _rebuild_target(source: Any, sql: str): - target = replace(source) - for action in _alter_clauses(_tokens(sql)): - words = tuple(token.upper() for token in action) - if words[:3] == ("MODIFY", "PARTITION", "BY"): - target = replace(target, partition_by=" ".join(action[3:])) - elif words[:3] == ("MODIFY", "ORDER", "BY"): - target = replace(target, order_by=_order_expressions(action[3:])) - elif words[:2] == ("MODIFY", "ENGINE"): - target = replace( - target, engine="".join(action[3:] if action[2:3] == ("=",) else action[2:]) - ) - return target - - -def _render_statement(console: Console, statement: dict[str, Any]) -> None: - classification = statement["classification"] - console.print(f" {statement['file']}:{statement['line']}") - console.print(" " + statement["sql"]) - console.print( - f" {classification['kind']} / {classification['table']}: {classification['detail']}" - ) - size = statement["size"] - if size: - prefix = "exact" if size["precision"] == "exact" else "ceiling: up to" - console.print( - f" {prefix} {size['compressed_bytes']} compressed bytes, {size['uncompressed_bytes']} uncompressed bytes, {size['part_count']} parts" - ) - console.print( - f" Columns: {', '.join(size['columns']) or 'all'}; partition ID: {size['partition_id'] or 'all'}" - ) - for node in statement["downstream"]: - console.print(f" Downstream {node['type']}: {node['name']}") - if statement["rebuild"]: - _render_rebuild(console, statement["rebuild"]) - if command := statement.get("suggested_command"): - console.print(" Suggested: " + command) - for finding in statement["findings"]: - _render_finding(console, finding) - - -def _render_rebuild(console: Console, rebuild: dict[str, Any]) -> None: - console.print( - f" Rebuild: {rebuild['bytes_on_disk']} bytes on disk; {rebuild['part_count']} parts" - ) - console.print(f" Parts per partition: {rebuild['partition_parts']}") - console.print(f" Insert rate: {rebuild['insert_rows_per_second']} rows/second") - console.print(" " + rebuild["engine_note"]) - for finding in rebuild["findings"]: - console.print( - f" {finding['severity']} [{finding['code']}]: {finding['message']} {finding['details']}" - ) - - -def _render_finding(console: Console, finding: dict[str, Any]) -> None: - console.print( - f" {finding['severity']} [{finding['rule']}] {finding['file']}:{finding['line']}: {finding['message']}" - ) - console.print(f" Blocking: {finding['blocking']}; waiver: {finding['waived']}") diff --git a/ch_migrate/rebuild.py b/ch_migrate/rebuild.py deleted file mode 100644 index 770a472..0000000 --- a/ch_migrate/rebuild.py +++ /dev/null @@ -1,168 +0,0 @@ -"""Alembic operation entry point for a resumable online table rebuild.""" - -from __future__ import annotations - -import hashlib -import json -from dataclasses import asdict -from functools import partial - -from alembic.operations import MigrateOperation, Operations - -from ch_migrate.classify import _identifier, _tokens -from ch_migrate.connection import get_client -from ch_migrate.introspect import get_create_statement, parse_create_table -from ch_migrate.rebuild_ddl import build_definition -from ch_migrate.rebuild_types import DEFAULT_GRACE_SECONDS, RebuildOptions, RebuildRuntime -from ch_migrate.sql import load_statements -from ch_migrate.waiting_types import WaitingError - -_OPTIONS = { - "select", - "allow_unacknowledged_async_loss", - "writers_on_this_service_only", - "grace_seconds", -} - - -@Operations.register_operation("rebuild_table") -class RebuildTableOp(MigrateOperation): - """One logical journal entry; internal writes use the rebuild recovery machine.""" - - def __init__(self, table: str, create_sql_path: str, options: dict): - self.table = table - self.create_sql_path = create_sql_path - self.options = options - - @classmethod - def rebuild_table(cls, operations, table, create_sql_path, **options): - unknown = set(options) - _OPTIONS - if unknown: - raise TypeError(f"rebuild_table got unknown options: {sorted(unknown)}") - return operations.invoke(cls(table, create_sql_path, options)) - - -def rebuild_table( - table, - create_sql_path, - *, - select=None, - allow_unacknowledged_async_loss=False, - writers_on_this_service_only=False, - grace_seconds=DEFAULT_GRACE_SECONDS, -): - """Rebuild from one replacement CREATE TABLE in migrations/sql/. - - On ClickHouse Cloud, `writers_on_this_service_only=True` declares that every writer to - the table uses this service; ch-migrate cannot see other services' inserts. - """ - from alembic import op - - return op.rebuild_table( - table, - create_sql_path, - select=select, - allow_unacknowledged_async_loss=allow_unacknowledged_async_loss, - writers_on_this_service_only=writers_on_this_service_only, - grace_seconds=grace_seconds, - ) - - -@Operations.implementation_for(RebuildTableOp) -def execute_rebuild_operation(operations: Operations, operation: RebuildTableOp): - from ch_migrate.rebuild_engine import execute_rebuild - - opts = operations.get_context().opts - waiter = opts.get("ch_migrate_waiter") - if waiter is None or waiter.revision is None: - raise WaitingError( - "rebuild_table requires a ch-migrate upgrade with the current environment" - ) - statements = load_statements(operation.create_sql_path) - if len(statements) != 1: - raise ValueError("rebuild_table requires exactly one replacement CREATE TABLE statement") - sql = statements[0].sql - request = _request(operation, waiter) - key, record, _ = waiter.reserve( - {"kind": "rebuild", "phase": "intent", "digest": _digest(request, sql), "repeat_safe": True} - ) - source = _source(waiter, request, record) - definition = build_definition(source, sql, request) - record.setdefault("source_ddl", source.raw_ddl) - runtime = RebuildRuntime( - client=waiter.client, - control_factory=partial(get_client, opts["ch_migrate_env_config"]), - definition=definition, - deployment=waiter.state, - budget=waiter.budget, - record=record, - checkpoint=lambda: waiter.journal.write(key, record), - refresh=lambda: _refresh(waiter, key, record), - ) - result = execute_rebuild(runtime) - record.update(phase="done", result=result) - runtime.checkpoint() - return result - - -def _request(operation: RebuildTableOp, waiter) -> RebuildOptions: - if not isinstance(operation.table, str) or not operation.table: - raise ValueError("rebuild_table requires a nonempty table name") - tokens = _tokens(operation.table) - database = waiter.state.database - if len(tokens) == 1: - table = _identifier(tokens[0]) - elif len(tokens) == 3 and tokens[1] == "." and _identifier(tokens[0]) == database: - table = _identifier(tokens[2]) - else: - raise ValueError("rebuild_table requires a table in the migration database") - return RebuildOptions( - database=database, - table=table, - revision=waiter.revision, - generation=waiter.generation, - cluster=waiter.state.on_cluster, - **_options(operation.options), - ) - - -def _options(given: dict) -> dict: - options = { - "select": None, - "allow_unacknowledged_async_loss": False, - "writers_on_this_service_only": False, - "grace_seconds": DEFAULT_GRACE_SECONDS, - **given, - } - for name in ("allow_unacknowledged_async_loss", "writers_on_this_service_only"): - if not isinstance(options[name], bool): - raise ValueError(f"{name} must be an explicit boolean") - grace = options["grace_seconds"] - if isinstance(grace, bool) or not isinstance(grace, (int, float)) or grace < 0: - raise ValueError("grace_seconds must be a non-negative number") - return options - - -def _digest(options: RebuildOptions, sql: str) -> str: - payload = {"options": asdict(options), "sql": sql} - return hashlib.sha256(json.dumps(payload, sort_keys=True).encode()).hexdigest() - - -def _source(waiter, request: RebuildOptions, record: dict): - ddl = record.get("source_ddl") - if ddl is None: - ddl = get_create_statement(waiter.client, request.database, request.table) - source = parse_create_table(ddl) - if source is None: - raise ValueError("Cannot parse the rebuild source's CREATE TABLE") - return source - - -def _refresh(waiter, key, current): - # Ownership may have changed since the CLI's initial journal snapshot. - # Read the previous owner's acknowledged checkpoints on this replica. - waiter.journal.synchronize() - previous = waiter.journal.read(key) - if previous and previous.get("phase") == "rejected" and current.get("phase") == "intent": - return None # Known pre-copy rejection was cleaned; reserve authorized a fresh attempt. - return previous diff --git a/ch_migrate/rebuild_ddl.py b/ch_migrate/rebuild_ddl.py deleted file mode 100644 index 8c2a23b..0000000 --- a/ch_migrate/rebuild_ddl.py +++ /dev/null @@ -1,625 +0,0 @@ -"""Build exact-schema helper DDL for an online ClickHouse table rebuild.""" - -from __future__ import annotations - -import re -from dataclasses import dataclass - -from ch_migrate.introspect import ColumnDefinition, TableDefinition -from ch_migrate.rebuild_types import HelperIdentity, RebuildDefinition, RebuildOptions - -_ENGINE = re.compile( - r"(?:(?:Replicated|Shared))?(?:Replacing|Summing|Aggregating|Collapsing|VersionedCollapsing|Graphite)?MergeTree\Z", - re.I, -) -_IDENTIFIER = re.compile(r"[A-Za-z_][A-Za-z_0-9]*\Z") -_CLAUSES = {"ENGINE", "PARTITION", "PRIMARY", "ORDER", "SAMPLE", "TTL", "SETTINGS", "COMMENT"} -_COLUMN_EXTRAS = {"INDEX", "PROJECTION", "CONSTRAINT"} -_COLUMN_MODIFIERS = {"DEFAULT", "MATERIALIZED", "ALIAS", "CODEC", "COMMENT", "TTL"} -_PLAIN_ROLES = {"snap", "stage"} - - -def build_definition( - source: TableDefinition, requested_sql: str, options: RebuildOptions -) -> RebuildDefinition: - """Validate the one explicit target DDL and retain its complete SQL text.""" - if source.name != options.table or not source.raw_ddl: - raise ValueError("source name or SHOW CREATE definition does not match rebuild table") - target_ddl = _parse(requested_sql, options.database, options.table) - _parse(source.raw_ddl, options.database, options.table) - if target_ddl.uuid_span: - raise ValueError("target CREATE must not specify a UUID; rebuild owns helper UUIDs") - if target_ddl.cluster_span: - raise ValueError("target CREATE must not specify ON CLUSTER; deployment selects it") - columns = tuple( - column.name - for column in target_ddl.columns - if column.default_kind not in {"MATERIALIZED", "ALIAS"} - ) - if not columns: - raise ValueError("target needs at least one insertable column") - projection = _projection(options.select, columns) - target = TableDefinition( - name=options.table, - engine=_engine_text(target_ddl), - columns=list(target_ddl.columns), - order_by=_order_columns(target_ddl.clauses.get("ORDER", "")), - partition_by=target_ddl.clauses.get("PARTITION"), - ttl=target_ddl.clauses.get("TTL"), - settings=_settings(target_ddl.clauses.get("SETTINGS", "")), - raw_ddl=target_ddl.sql, - ) - return RebuildDefinition( - database=options.database, - table=options.table, - source=source, - target=target, - create_sql=target_ddl.sql, - projection=projection, - columns=columns, - cluster=options.cluster, - revision=options.revision, - generation=options.generation, - allow_unacknowledged_async_loss=options.allow_unacknowledged_async_loss, - writers_on_this_service_only=options.writers_on_this_service_only, - grace_seconds=options.grace_seconds, - ) - - -def helper_ddl(definition: RebuildDefinition, role: str, identity: HelperIdentity) -> str: - """Create a same-database physical helper that the rebuild can prove is its own. - - Self-hosted helpers get a chosen UUID and their own Keeper path. On ClickHouse Cloud, - which rejects chosen UUIDs, they carry the owner token as their COMMENT instead. - `snap` and `stage` are plain MergeTree without TTL, in the source's replication - family: their merges keep every row, so snapshot counts hold without - SYSTEM STOP MERGES, which a server restart silently undoes. - """ - if role not in {"new", "snap", "stage"}: - raise ValueError(f"unknown rebuild helper role: {role}") - if identity.token is None: - _validate_uuid(identity.uuid) - schema = definition.source if role == "snap" else definition.target - parsed = _parse(schema.raw_ddl, definition.database, definition.table) - name = f"{definition.table}__chm_{role}" - replacements = [ - ( - parsed.tokens[parsed.name_at].start, - parsed.tokens[parsed.name_at + 2].end, - f"{_quote(definition.database)}.{_quote(name)}", - ), - (parsed.engine_at, parsed.engine_end, _helper_engine(parsed, role, identity)), - *_identity_replacements(parsed, identity), - ] - if role in _PLAIN_ROLES: - replacements.extend((start, end, "") for start, end in _ttl_spans(parsed)) - if parsed.cluster_span: - replacements.append((*parsed.cluster_span, "")) - sql = _replace(parsed.sql, replacements) - if definition.cluster: - sql = _insert_cluster(sql, definition.cluster) - return sql - - -def dual_ddl(definition: RebuildDefinition, identity: HelperIdentity) -> str: - """Mirror incoming source inserts through a view the rebuild can prove is its own.""" - if identity.token is None: - _validate_uuid(identity.uuid) - marker, comment = f" UUID '{identity.uuid}'", "" - else: - # COMMENT before AS: after the SELECT it would parse as a table alias. - marker, comment = "", f" COMMENT {_string_literal(identity.token)}" - db = _quote(definition.database) - table = _quote(definition.table) - cluster = f" ON CLUSTER {_quote(definition.cluster)}" if definition.cluster else "" - return ( - f"CREATE MATERIALIZED VIEW {db}.{_quote(definition.table + '__chm_dual')}" - f"{marker}{cluster} TO {db}.{_quote(definition.table + '__chm_new')}{comment} " - f"AS SELECT {definition.projection} FROM {db}.{table}" - ) - - -def target_comment(definition: RebuildDefinition) -> tuple[str, str]: - """The replacement's own COMMENT, as an ALTER for `new` and as the expected value.""" - parsed = _parse(definition.target.raw_ddl, definition.database, definition.table) - literal = parsed.clauses.get("COMMENT", "''") - if not _string(literal): - raise ValueError("target COMMENT must be a string literal") - name = f"{_quote(definition.database)}.{_quote(definition.table + '__chm_new')}" - return f"ALTER TABLE {name} MODIFY COMMENT {literal}", _unquote(literal) - - -@dataclass(frozen=True) -class _Token: - text: str - start: int - end: int - - -@dataclass(frozen=True) -class _DDL: - sql: str - tokens: tuple[_Token, ...] - name_at: int - schema_at: int - schema_end: int - engine_at: int - engine_end: int - engine_name: str - engine_args: tuple[str, ...] | None - header_end: int - uuid_span: tuple[int, int] | None - cluster_span: tuple[int, int] | None - columns: tuple[ColumnDefinition, ...] - clauses: dict[str, str] - - -def _parse(sql: str, database: str, table: str) -> _DDL: - text = sql.strip() - tokens = _tokens(text) - if tokens and tokens[-1].text == ";": - text = text[: tokens[-1].start].rstrip() - tokens = _tokens(text) - if any(token.text == ";" for token in tokens): - raise ValueError("rebuild SQL must contain exactly one CREATE TABLE") - if len(tokens) < 9 or _upper(tokens[0]) != "CREATE" or _upper(tokens[1]) != "TABLE": - raise ValueError("rebuild SQL must be one CREATE TABLE with an explicit schema") - if _upper(tokens[2]) in {"IF", "TEMPORARY"}: - raise ValueError("conditional or temporary CREATE TABLE is unsafe for rebuild") - if tokens[3].text != "." and _ident(tokens[2]) == table: - text = ( - text[: tokens[2].start] + f"{_quote(database)}.{tokens[2].text}" + text[tokens[2].end :] - ) - tokens = _tokens(text) - if _ident(tokens[2]) != database or tokens[3].text != "." or _ident(tokens[4]) != table: - raise ValueError("CREATE TABLE must target the requested database and table") - position, uuid_span, cluster_span = _header(tokens, 5) - if position >= len(tokens) or tokens[position].text != "(": - raise ValueError("CREATE TABLE requires an explicit column schema (never AS table)") - close = _closing(tokens, position) - columns = _columns(text, tokens, (position, close)) - clauses = _clauses(text, tokens, close + 1) - engine_at, engine_end, engine_name, args = _engine(text, tokens, close + 1) - if not _ENGINE.fullmatch(engine_name): - raise ValueError(f"unsupported rebuild table engine: {engine_name}") - return _DDL( - text, - tokens, - 2, - tokens[position].start, - tokens[close].end, - engine_at, - engine_end, - engine_name, - args, - tokens[position].start, - uuid_span, - cluster_span, - columns, - clauses, - ) - - -def _header( - tokens: tuple[_Token, ...], position: int -) -> tuple[int, tuple[int, int] | None, tuple[int, int] | None]: - uuid_span = cluster_span = None - while position < len(tokens) and tokens[position].text != "(": - word = _upper(tokens[position]) - if word == "UUID" and uuid_span is None and position + 1 < len(tokens): - if not _string(tokens[position + 1].text): - raise ValueError("invalid CREATE TABLE UUID") - uuid_span = (tokens[position].start, tokens[position + 1].end) - position += 2 - elif word == "ON" and cluster_span is None and position + 2 < len(tokens): - if _upper(tokens[position + 1]) != "CLUSTER": - raise ValueError("unsupported CREATE TABLE header") - cluster_span = (tokens[position].start, tokens[position + 2].end) - position += 3 - else: - raise ValueError("unsupported or ambiguous CREATE TABLE header") - return position, uuid_span, cluster_span - - -def _tokens(text: str) -> tuple[_Token, ...]: - result: list[_Token] = [] - index = 0 - while index < len(text): - if text[index].isspace(): - index += 1 - continue - if text.startswith("--", index) or text.startswith("/*", index): - end = text.find("\n" if text.startswith("--", index) else "*/", index + 2) - if end == -1: - if text.startswith("/*", index): - raise ValueError("unterminated SQL comment") - break - index = end + (0 if text.startswith("--", index) else 2) - continue - start = index - quote = text[index] - if quote in {"'", '"', "`"}: - index = _quoted_end(text, index) - elif quote.isalpha() or quote == "_": - index += 1 - while index < len(text) and (text[index].isalnum() or text[index] == "_"): - index += 1 - else: - index += 1 - result.append(_Token(text[start:index], start, index)) - return tuple(result) - - -def _quoted_end(text: str, index: int) -> int: - quote = text[index] - index += 1 - while index < len(text): - if text[index] == "\\": - index += 2 - elif text[index] == quote: - index += 1 - if index < len(text) and text[index] == quote: - index += 1 - else: - return index - else: - index += 1 - raise ValueError("unterminated SQL quote") - - -def _closing(tokens: tuple[_Token, ...], opening: int) -> int: - depth = 0 - for index in range(opening, len(tokens)): - if tokens[index].text == "(": - depth += 1 - elif tokens[index].text == ")": - depth -= 1 - if depth == 0: - return index - raise ValueError("unbalanced CREATE TABLE parentheses") - - -def _split(tokens: tuple[_Token, ...], start: int, end: int) -> list[tuple[int, int]]: - groups: list[tuple[int, int]] = [] - depth = 0 - beginning = start - for index in range(start, end): - if tokens[index].text in ("(", "[", "{"): - depth += 1 - elif tokens[index].text in (")", "]", "}"): - depth -= 1 - elif tokens[index].text == "," and depth == 0: - groups.append((beginning, index)) - beginning = index + 1 - groups.append((beginning, end)) - if depth or any(a == b for a, b in groups): - raise ValueError("ambiguous or empty schema expression") - return groups - - -def _columns( - text: str, tokens: tuple[_Token, ...], bounds: tuple[int, int] -) -> tuple[ColumnDefinition, ...]: - start, end = bounds - columns: list[ColumnDefinition] = [] - for first, last in _split(tokens, start + 1, end): - if _upper(tokens[first]) in _COLUMN_EXTRAS: - continue - if last - first < 2: - raise ValueError("column definition needs a name and type") - columns.append(_column(text, tokens[first:last])) - if not columns or len({column.name for column in columns}) != len(columns): - raise ValueError("CREATE TABLE has no columns or duplicate columns") - return tuple(columns) - - -def _column(text: str, tokens: tuple[_Token, ...]) -> ColumnDefinition: - name = _ident(tokens[0]) - if not name: - raise ValueError("unrecognized column definition") - modifiers: list[tuple[str, int]] = [] - depth = 0 - for index, token in enumerate(tokens[1:], 1): - if token.text in ("(", "[", "{"): - depth += 1 - elif token.text in (")", "]", "}"): - depth -= 1 - if depth == 0 and _upper(token) in _COLUMN_MODIFIERS: - modifiers.append((_upper(token), index)) - type_end = modifiers[0][1] if modifiers else len(tokens) - column_type = text[tokens[1].start : tokens[type_end - 1].end] - values = {} - for offset, (kind, index) in enumerate(modifiers): - end = modifiers[offset + 1][1] if offset + 1 < len(modifiers) else len(tokens) - values[kind] = text[tokens[index + 1].start : tokens[end - 1].end] - kind = next((item for item in ("MATERIALIZED", "ALIAS", "DEFAULT") if item in values), None) - return ColumnDefinition( - name, column_type, kind, values.get(kind), values.get("CODEC"), values.get("COMMENT") - ) - - -def _clauses(text: str, tokens: tuple[_Token, ...], start: int) -> dict[str, str]: - starts: list[tuple[str, int, int]] = [] - position = start - while position < len(tokens): - word = _upper(tokens[position]) - if word not in _CLAUSES: - raise ValueError(f"unsupported CREATE TABLE clause: {tokens[position].text}") - next_position = position + (2 if word in {"PARTITION", "PRIMARY", "ORDER", "SAMPLE"} else 1) - expected = "KEY" if word == "PRIMARY" else "BY" - if next_position > len(tokens) or ( - next_position == position + 2 and _upper(tokens[position + 1]) != expected - ): - raise ValueError(f"invalid {word} clause") - if any(name == word for name, _, _ in starts): - raise ValueError(f"duplicate {word} clause") - starts.append((word, position, next_position)) - position = next_position - depth = 0 - while position < len(tokens): - token = tokens[position] - if token.text == "(": - depth += 1 - elif token.text == ")": - depth -= 1 - if depth == 0 and _upper(token) in _CLAUSES and _clause_start(tokens, position): - break - position += 1 - if depth or position == next_position: - raise ValueError(f"unbalanced or empty {word} clause") - clauses: dict[str, str] = {} - for index, (word, _, value_start) in enumerate(starts): - stop = starts[index + 1][1] if index + 1 < len(starts) else len(tokens) - clauses[word] = text[tokens[value_start].start : tokens[stop - 1].end] - if "ENGINE" not in clauses or "ORDER" not in clauses: - raise ValueError("rebuild requires ENGINE and ORDER BY") - return clauses - - -def _clause_start(tokens: tuple[_Token, ...], index: int) -> bool: - word = _upper(tokens[index]) - return word not in {"PARTITION", "PRIMARY", "ORDER", "SAMPLE"} or ( - index + 1 < len(tokens) - and _upper(tokens[index + 1]) == ("KEY" if word == "PRIMARY" else "BY") - ) - - -def _engine( - text: str, tokens: tuple[_Token, ...], start: int -) -> tuple[int, int, str, tuple[str, ...] | None]: - depth = 0 - for index in range(start, len(tokens)): - token = tokens[index] - if token.text == "(": - depth += 1 - elif token.text == ")": - depth -= 1 - if depth or _upper(token) != "ENGINE": - continue - if index + 2 >= len(tokens) or tokens[index + 1].text != "=": - raise ValueError("ENGINE must use explicit ENGINE = name syntax") - name = _ident(tokens[index + 2]) - if not name: - raise ValueError("invalid ENGINE name") - end = index + 2 - args = None - if end + 1 < len(tokens) and tokens[end + 1].text == "(": - close = _closing(tokens, end + 1) - args = ( - tuple( - text[tokens[a].start : tokens[b - 1].end] - for a, b in _split(tokens, end + 2, close) - ) - if close > end + 2 - else () - ) - end = close - return tokens[index].start, tokens[end].end, name, args - raise ValueError("missing ENGINE clause") - - -def _helper_engine(parsed: _DDL, role: str, identity: HelperIdentity) -> str: - name = parsed.engine_name - args = parsed.engine_args or () - prefix = re.match(r"(?:replicated|shared)?", name, re.I).group(0) - if identity.token is not None: - # Cloud chooses Shared storage and Keeper paths itself and rejects explicit ones. - keeper, params = (), args[2:] if prefix and _keeper_given(args) else args - name, prefix = name[len(prefix) :], "" - elif prefix: - keeper, params = _keeper_arguments(parsed, role, identity.uuid) - else: - keeper, params = (), args - if role in _PLAIN_ROLES: - name, params = prefix + "MergeTree", () - args = (*keeper, *params) - return ( - f"ENGINE = {name}({', '.join(args)})" - if args or parsed.engine_args is not None - else f"ENGINE = {name}" - ) - - -def _identity_replacements(parsed: _DDL, identity: HelperIdentity) -> list[tuple[int, int, str]]: - """A chosen UUID clause, or on Cloud no UUID and the owner token as the table COMMENT.""" - if identity.token is None: - if parsed.uuid_span: - return [(*parsed.uuid_span, f"UUID '{identity.uuid}'")] - return [(parsed.schema_at, parsed.schema_at, f"UUID '{identity.uuid}' ")] - removed = [(*parsed.uuid_span, "")] if parsed.uuid_span else [] - comment = f"COMMENT {_string_literal(identity.token)}" - closing = _closing(parsed.tokens, _schema_opening(parsed)) - clauses = _keyword_spans(parsed.tokens, (closing + 1, len(parsed.tokens)), _CLAUSES) - if "COMMENT" in clauses: - return removed + [(*clauses["COMMENT"], comment)] - return removed + [(len(parsed.sql), len(parsed.sql), " " + comment)] - - -def _keeper_given(args: tuple[str, ...]) -> bool: - return len(args) >= 2 and _string(args[0]) and _string(args[1]) - - -def _keeper_arguments( - parsed: _DDL, role: str, uuid: str -) -> tuple[tuple[str, str], tuple[str, ...]]: - """The helper's own Keeper path and replica, and the engine's remaining parameters.""" - args = parsed.engine_args or () - if args and _string(args[0]) and (len(args) < 2 or not _string(args[1])): - raise ValueError("ambiguous replicated Keeper path/replica arguments") - if len(args) < 2 or not _string(args[0]): - return (f"'/clickhouse/tables/{uuid}/{{shard}}'", "'{replica}'"), args - original = ( - _unquote(args[0]) - .replace("{database}", _ident(parsed.tokens[parsed.name_at])) - .replace("{table}", _ident(parsed.tokens[parsed.name_at + 2])) - ) - # Sibling roots survive removal of the old table's Keeper subtree. - # Freeze name-dependent macros before the eventual EXCHANGE/RENAME. - parent = original.rstrip("/").rsplit("/", 1)[0] - path = parent + f"/__chm_{role}_{uuid}" - for macro in dict.fromkeys(re.findall(r"\{[A-Za-z_][A-Za-z_0-9]*\}", original)): - if macro not in path: - path += "/" + macro # Retain shard/UUID discrimination from the old leaf. - return ("'" + path.replace("'", "\\'") + "'", args[1]), args[2:] - - -def _ttl_spans(parsed: _DDL) -> list[tuple[int, int]]: - """Every table and column TTL, which a plain helper drops so merges keep every row.""" - tokens = parsed.tokens - opening = _schema_opening(parsed) - closing = _closing(tokens, opening) - groups = [ - _keyword_spans(tokens, (first + 1, last), _COLUMN_MODIFIERS) - for first, last in _split(tokens, opening + 1, closing) - if _upper(tokens[first]) not in _COLUMN_EXTRAS - ] - groups.append(_keyword_spans(tokens, (closing + 1, len(tokens)), _CLAUSES)) - return [group["TTL"] for group in groups if "TTL" in group] - - -def _schema_opening(parsed: _DDL) -> int: - return next(i for i, token in enumerate(parsed.tokens) if token.start == parsed.schema_at) - - -def _keyword_spans( - tokens: tuple[_Token, ...], bounds: tuple[int, int], keywords: set[str] -) -> dict[str, tuple[int, int]]: - """Each depth-0 keyword's span, up to the next depth-0 keyword or the bounds' end.""" - starts, depth = [], 0 - for index in range(*bounds): - token = tokens[index] - depth += token.text in ("(", "[", "{") - depth -= token.text in (")", "]", "}") - if not depth and _upper(token) in keywords and _clause_start(tokens, index): - starts.append(index) - ends = [index - 1 for index in starts[1:]] + [bounds[1] - 1] - return { - _upper(tokens[start]): (tokens[start].start, tokens[end].end) - for start, end in zip(starts, ends) - } - - -def _insert_cluster(sql: str, cluster: str) -> str: - tokens = _tokens(sql) - offset = tokens[6].end if len(tokens) > 6 and _upper(tokens[5]) == "UUID" else tokens[4].end - return sql[:offset] + f" ON CLUSTER {_quote(cluster)}" + sql[offset:] - - -def _projection(select: str | None, columns: tuple[str, ...]) -> str: - if select is None: - return ", ".join(_quote(column) for column in columns) - tokens = _tokens(select) - forbidden = { - "SELECT", - "FROM", - "JOIN", - "WHERE", - "UNION", - "INTO", - "PREWHERE", - "GROUP", - "HAVING", - "LIMIT", - } - if not tokens or any(_upper(token) in forbidden or token.text == ";" for token in tokens): - raise ValueError("select must be a projection expression list, not a full query") - expressions = _split(tokens, 0, len(tokens)) - if len(expressions) != len(columns): - raise ValueError("projection expression count must match insertable target columns") - projected = [] - for (first, last), column in zip(expressions, columns): - end = last - if last - first >= 3 and _upper(tokens[last - 2]) == "AS": - if _ident(tokens[last - 1]) != column: - raise ValueError("projection alias must equal the corresponding target column") - end = last - 2 - expr = select[tokens[first].start : tokens[end - 1].end] - projected.append(f"({expr}) AS {_quote(column)}") - return ", ".join(projected) - - -def _order_columns(expression: str) -> list[str]: - tokens = _tokens(expression) - if tokens and tokens[0].text == "(" and _closing(tokens, 0) == len(tokens) - 1: - tokens = tokens[1:-1] - return ( - [expression[tokens[a].start : tokens[b - 1].end] for a, b in _split(tokens, 0, len(tokens))] - if tokens - else [] - ) - - -def _settings(expression: str) -> dict[str, str]: - tokens = _tokens(expression) - if not tokens: - return {} - result = {} - for first, last in _split(tokens, 0, len(tokens)): - if last - first < 3 or tokens[first + 1].text != "=": - raise ValueError("invalid ENGINE settings") - result[_ident(tokens[first])] = expression[tokens[first + 2].start : tokens[last - 1].end] - return result - - -def _engine_text(parsed: _DDL) -> str: - return parsed.sql[parsed.engine_at : parsed.engine_end].split("=", 1)[1].strip() - - -def _replace(sql: str, replacements: list[tuple[int, int, str]]) -> str: - for start, end, replacement in sorted(replacements, reverse=True): - sql = sql[:start] + replacement + sql[end:] - return sql - - -def _ident(token: _Token) -> str: - text = token.text - if text.startswith(("`", '"')) and text.endswith(text[0]): - return text[1:-1].replace(text[0] * 2, text[0]) - return text if _IDENTIFIER.fullmatch(text) else "" - - -def _upper(token: _Token) -> str: - return token.text.upper() if _IDENTIFIER.fullmatch(token.text) else "" - - -def _string(text: str) -> bool: - return len(text) >= 2 and text[0] == "'" and text[-1] == "'" - - -def _unquote(text: str) -> str: - return text[1:-1].replace("\\'", "'").replace("''", "'") - - -def _string_literal(value: str) -> str: - return "'" + value.replace("\\", "\\\\").replace("'", "\\'") + "'" - - -def _quote(identifier: str) -> str: - if not identifier or "\x00" in identifier: - raise ValueError("invalid SQL identifier") - return "`" + identifier.replace("`", "``") + "`" - - -def _validate_uuid(uuid: str) -> None: - if not re.fullmatch(r"[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}", uuid): - raise ValueError("helper UUID must be a canonical UUID") diff --git a/ch_migrate/rebuild_dependents.py b/ch_migrate/rebuild_dependents.py deleted file mode 100644 index a6c6f45..0000000 --- a/ch_migrate/rebuild_dependents.py +++ /dev/null @@ -1,382 +0,0 @@ -"""Read-only dependency inspection and pre-copy validation for online rebuilds.""" - -from __future__ import annotations - -from dataclasses import dataclass -from typing import Any - -from ch_migrate.introspect import get_live_schema, parse_create_dictionary -from ch_migrate.rebuild_ddl import _closing, _ident, _quote, _replace, _tokens, _upper -from ch_migrate.rebuild_types import RebuildDefinition -from ch_migrate.waiting_sql import sql_string -from ch_migrate.waiting_types import WaitingError - - -class DependentValidationError(WaitingError): - """A named dependent cannot safely use the replacement table.""" - - -@dataclass(frozen=True) -class DependentQuery: - name: str - sql: str - kind: str - - -@dataclass(frozen=True) -class DependentInventory: - source_views: tuple[DependentQuery, ...] - target_views: tuple[DependentQuery, ...] - dictionaries: tuple[DependentQuery, ...] - - -@dataclass(frozen=True) -class DictionaryReload: - database: str - names: tuple[str, ...] - cluster: str | None = None - - -def inspect_dependents(client: Any, database: str, table: str) -> DependentInventory: - """Find only same-database objects actually reading or writing this table.""" - schema = get_live_schema(client, database) - source: list[DependentQuery] = [] - target: list[DependentQuery] = [] - dictionaries: list[DependentQuery] = [] - for name, view in schema.views.items(): - if view.select_query and _references(view.select_query, database, table): - source.append(DependentQuery(name, view.select_query, "view")) - for name, view in schema.materialized_views.items(): - if not view.select_query: - continue - query = DependentQuery(name, view.select_query, "materialized_view") - if _references(query.sql, database, table): - source.append(query) - if _same_object(view.target_table, database, table): - target.append(query) - for name, dictionary in _dictionary_definitions(client, schema, database).items(): - query = _dictionary_query(dictionary, database, table) - if query is not None: - dictionaries.append(DependentQuery(name, query, "dictionary")) - return DependentInventory(tuple(source), tuple(target), tuple(dictionaries)) - - -def validate_dependents( - client: Any, definition: RebuildDefinition, inventory: DependentInventory -) -> None: - """Resolve each dependent against the owned replacement without changing data.""" - database, table = definition.database, definition.table - replacement = table + "__chm_new" - seen: set[str] = set() - for dependent in (*inventory.source_views, *inventory.target_views, *inventory.dictionaries): - try: - rewritten, count = _rewrite(dependent.sql, _RewriteTarget(database, table, replacement)) - if dependent in inventory.source_views or dependent in inventory.dictionaries: - if count == 0: - raise ValueError("source reference to rebuilt table not found") - if dependent.kind == "dictionary": - _validate_dictionary( - client, f"{_quote(database)}.{_quote(dependent.name)}", rewritten - ) - else: - if dependent.name not in seen: - client.query("EXPLAIN PLAN " + rewritten) - _header(client, rewritten) - seen.add(dependent.name) - if dependent in inventory.target_views: - _validate_target(client, f"{_quote(database)}.{_quote(replacement)}", rewritten) - except Exception as exc: - if isinstance(exc, DependentValidationError): - raise - raise DependentValidationError( - f"Cannot validate {dependent.kind} {database}.{dependent.name}: {exc}" - ) from exc - - -def reload_dictionaries(client: Any, request: DictionaryReload) -> None: - """Reload every affected dictionary, propagating its named failure.""" - for name in request.names: - try: - cluster = f" ON CLUSTER {_quote(request.cluster)}" if request.cluster else "" - client.command( - f"SYSTEM RELOAD DICTIONARY{cluster} " f"{_quote(request.database)}.{_quote(name)}" - ) - except Exception as exc: - raise DependentValidationError( - f"Cannot reload dictionary {request.database}.{name}: {exc}" - ) from exc - - -@dataclass(frozen=True) -class _RewriteTarget: - database: str - table: str - replacement: str - - -def _dictionary_definitions(client: Any, schema: Any, database: str) -> dict: - """system.dictionaries can be empty even when system.tables has dictionaries.""" - dictionaries = dict(schema.dictionaries) - rows = client.query( - "SELECT name FROM system.tables WHERE database = {db:String} " "AND engine = 'Dictionary'", - parameters={"db": database}, - ).result_rows - for (name,) in rows: - if name not in dictionaries: - ddl = client.query( - f"SHOW CREATE DICTIONARY {_quote(database)}.{_quote(name)}" - ).result_rows[0][0] - parsed = parse_create_dictionary(ddl) - if parsed is None: - raise DependentValidationError(f"Cannot parse dictionary {database}.{name}") - dictionaries[name] = parsed - return dictionaries - - -def _dictionary_query(dictionary: Any, database: str, table: str) -> str | None: - ddl = dictionary.raw_ddl - tokens = _tokens(ddl) - source = next((i for i, t in enumerate(tokens) if _upper(t) == "SOURCE"), None) - if source is None or source + 3 >= len(tokens): - raise DependentValidationError( - f"Cannot inspect dictionary {database}.{dictionary.name} SOURCE" - ) - if _upper(tokens[source + 2]) != "CLICKHOUSE": - return None - options = _source_options(tokens, source + 4) - host = options.get("HOST", "localhost").lower() - if host not in {"localhost", "127.0.0.1", "::1"}: - return None - source_db = options.get("DB") - source_table = options.get("TABLE") - if source_table and "." in source_table: - source_db, source_table = source_table.rsplit(".", 1) - if source_db != database: - return None - if "QUERY" in options: - query = options["QUERY"] - return query if _references(query, database, table) else None - if source_table != table: - return None - columns = _dictionary_columns(tokens) - if not columns: - raise DependentValidationError( - f"Cannot inspect dictionary {database}.{dictionary.name} columns" - ) - return ( - f"SELECT {', '.join(_quote(name) for name in columns)} " - f"FROM {_quote(database)}.{_quote(table)}" - ) - - -def _source_options(tokens: tuple, start: int) -> dict[str, str]: - options: dict[str, str] = {} - depth = 0 - for index in range(start, len(tokens) - 1): - text = tokens[index].text - if text == "(": - depth += 1 - elif text == ")": - if depth == 0: - break - depth -= 1 - elif depth == 0 and _upper(tokens[index]) and tokens[index + 1].text.startswith("'"): - value = tokens[index + 1].text[1:-1] - options[_upper(tokens[index])] = value.replace("\\'", "'").replace("''", "'") - return options - - -def _dictionary_columns(tokens: tuple) -> tuple[str, ...]: - opening = next((i for i, token in enumerate(tokens) if token.text == "("), None) - if opening is None: - return () - depth = 0 - names: list[str] = [] - for index in range(opening + 1, len(tokens)): - token = tokens[index] - if token.text == "(": - depth += 1 - elif token.text == ")": - if depth == 0: - break - depth -= 1 - elif token.text == "," and depth == 0: - continue - elif (index == opening + 1 or tokens[index - 1].text == ",") and depth == 0: - names.append(_ident(token)) - return tuple(name for name in names if name) - - -def _same_object(reference: str | None, database: str, table: str) -> bool: - if reference is None: - return False - parts = reference.replace("`", "").split(".") - return parts == [table] or parts == [database, table] - - -def _references(sql: str, database: str, table: str) -> bool: - return _rewrite(sql, _RewriteTarget(database, table, table + "__chm_new"))[1] > 0 - - -def _rewrite(sql: str, target: _RewriteTarget) -> tuple[str, int]: - """Change FROM/JOIN object references, never text or arbitrary identifiers.""" - database, table, replacement = target.database, target.table, target.replacement - tokens = _tokens(sql) - changes: list[tuple[int, int, str]] = [] - for index, token in enumerate(tokens[:-1]): - if _upper(token) not in {"FROM", "JOIN"}: - continue - first = index + 1 - if tokens[first].text == "(": - continue - if first + 2 < len(tokens) and tokens[first + 1].text == ".": - db, name = _ident(tokens[first]), _ident(tokens[first + 2]) - if db == database and name == table: - changes.append( - ( - tokens[first].start, - tokens[first + 2].end, - f"{_quote(database)}.{_quote(replacement)}", - ) - ) - elif _ident(tokens[first]) == table and ( - first + 1 == len(tokens) or tokens[first + 1].text != "(" - ): - changes.append( - ( - tokens[first].start, - tokens[first].end, - f"{_quote(database)}.{_quote(replacement)}", - ) - ) - references = len(changes) - if references: - changes.extend(_column_qualifiers(tokens, target, _relation_aliases(tokens))) - return _replace(sql, changes), references - - -def _column_qualifiers(tokens: tuple, target: _RewriteTarget, aliases: set[str]) -> list: - changes = [] - for index, token in enumerate(tokens[:-1]): - if ( - index + 3 < len(tokens) - and _ident(token) == target.database - and tokens[index + 1].text == "." - and _ident(tokens[index + 2]) == target.table - and tokens[index + 3].text == "." - ): - changes.append( - ( - token.start, - tokens[index + 2].end, - f"{_quote(target.database)}.{_quote(target.replacement)}", - ) - ) - elif ( - _ident(token) == target.table - and target.table not in aliases - and tokens[index + 1].text == "." - and (index == 0 or tokens[index - 1].text != ".") - ): - changes.append((token.start, token.end, _quote(target.replacement))) - return changes - - -def _relation_aliases(tokens: tuple) -> set[str]: - boundaries = { - "FINAL", - "SAMPLE", - "WHERE", - "PREWHERE", - "GROUP", - "ORDER", - "HAVING", - "LIMIT", - "OFFSET", - "SETTINGS", - "FORMAT", - "UNION", - "EXCEPT", - "INTERSECT", - "JOIN", - "INNER", - "LEFT", - "RIGHT", - "FULL", - "CROSS", - "ANY", - "ALL", - "ASOF", - "SEMI", - "ANTI", - "GLOBAL", - "ARRAY", - "ON", - "USING", - "WINDOW", - "QUALIFY", - } - aliases = set() - for index, token in enumerate(tokens[:-1]): - if _upper(token) not in {"FROM", "JOIN"}: - continue - end = index + 1 - if tokens[end].text == "(": - end = _closing(tokens, end) - else: - while end + 2 < len(tokens) and tokens[end + 1].text == ".": - end += 2 - if end + 1 < len(tokens) and tokens[end + 1].text == "(": - end = _closing(tokens, end + 1) - candidate = end + 1 - if candidate < len(tokens) and _upper(tokens[candidate]) == "AS": - candidate += 1 - if candidate < len(tokens) and _upper(tokens[candidate]) not in boundaries: - if alias := _ident(tokens[candidate]): - aliases.add(alias) - return aliases - - -def _header(client: Any, sql: str) -> dict[str, str]: - result = client.query(f"SELECT * FROM ({sql.rstrip().rstrip(';')}) LIMIT 0") - return { - name: getattr(column_type, "name", str(column_type)) - for name, column_type in zip(result.column_names, result.column_types) - } - - -def _validate_dictionary(client: Any, qualified_name: str, sql: str) -> None: - ddl = client.query(f"SHOW CREATE DICTIONARY {qualified_name}").result_rows[0][0] - tokens = _tokens(ddl) - columns = _dictionary_columns(tokens) - client.query("EXPLAIN PLAN " + sql) - actual = _header(client, sql) - missing = [name for name in columns if name not in actual] - if missing: - raise ValueError(f"missing dictionary key/attribute columns: {missing}") - expected = _header(client, f"SELECT * FROM {qualified_name}") - conversions = {name: kind for name, kind in expected.items() if actual[name] != kind} - _validate_casts(client, sql, conversions) - - -def _validate_target(client: Any, qualified_name: str, sql: str) -> None: - actual = _header(client, sql) - target = _header(client, f"SELECT * FROM {qualified_name}") - if not actual: - raise ValueError("TO view SELECT has no output columns") - for name, source_type in actual.items(): - target_type = target.get(name) - if target_type is None: - raise ValueError(f"TO target has no column {name}") - conversions = {name: target[name] for name, kind in actual.items() if target[name] != kind} - _validate_casts(client, sql, conversions) - - -def _validate_casts(client: Any, sql: str, conversions: dict[str, str]) -> None: - if not conversions: - return - projection = ", ".join( - f"CAST(__chm_input.{_quote(name)}, {sql_string(kind)}) AS {_quote(name)}" - for name, kind in conversions.items() - ) - client.query(f"EXPLAIN PLAN SELECT {projection} FROM ({sql}) AS __chm_input") diff --git a/ch_migrate/rebuild_engine.py b/ch_migrate/rebuild_engine.py deleted file mode 100644 index be8c003..0000000 --- a/ch_migrate/rebuild_engine.py +++ /dev/null @@ -1,1137 +0,0 @@ -"""Durable, operator-released ClickHouse table rebuild.""" - -from __future__ import annotations - -import sys -import threading -import time -from uuid import uuid4 - -from ch_migrate.rebuild_ddl import dual_ddl, helper_ddl, target_comment -from ch_migrate.rebuild_dependents import ( - DependentValidationError, - DictionaryReload, - inspect_dependents, - reload_dictionaries, - validate_dependents, -) -from ch_migrate.rebuild_preflight import RebuildRequest, inspect_rebuild -from ch_migrate.rebuild_types import HelperIdentity -from ch_migrate.waiting_ddl import DistributedDDLWaiter -from ch_migrate.waiting_types import UnknownOutcome, WaitingError - -_HEARTBEAT_SECONDS = 2 -_STALE_SECONDS = 8 -_MAX_COPY_MEMORY = 1_073_741_824 - - -class StrandedRows(WaitingError): - """Acknowledged rows reached only the rollback table; the operator must copy them.""" - - -def execute_rebuild(runtime) -> dict: - """Run a rebuild; a failed owner retains its lock for operator reconciliation.""" - ctx = _Context(runtime) - previous = runtime.record.get("rebuild") - if previous and previous.get("result") is not None: - _completed_without_lock(ctx) - return previous["result"] - if not previous: - _preflight(ctx) - rejected = None - with _Lock(ctx) as lock: - try: - result = _run_locked(ctx, lock) - except DependentValidationError as error: - _discard_validation(ctx, error) - rejected = error - if rejected is not None: - raise rejected - return result - - -class _Context: - def __init__(self, runtime): - self.runtime = runtime - self.client = runtime.client - self.definition = runtime.definition - self.database = runtime.definition.database - self.table = runtime.definition.table - self.cluster = runtime.deployment.replica_cluster - # Cloud's Shared databases reject chosen UUIDs, STOP MERGES and queue flushes. - self.shared = runtime.deployment.database_engine.startswith("Shared") - self.ddl_cluster = runtime.definition.cluster - self.budget = runtime.budget - self.waiter = DistributedDDLWaiter(self.client, self.budget) - self.hosts = self._hosts() - self.dependents = None - - @property - def state(self): - return self.runtime.record["rebuild"] - - def save(self): - self.runtime.checkpoint() - - def name(self, role): - return f"{self.table}__chm_{role}" - - def identity(self, role): - """How this rebuild creates `role` so that it can later prove the object is its own.""" - if self.shared: - return HelperIdentity(token=self.state["tokens"][role]) - return HelperIdentity( - uuid=self.dual_uuid() if role == "dual" else self.state["helpers"][role] - ) - - def dual_uuid(self): - return self.state.get("dual_uuid") - - def bound_uuid(self, role): - """The helper's UUID: chosen up front, or on Cloud read back once its token proves ownership.""" - uid = self.dual_uuid() if role == "dual" else self.state["helpers"][role] - if uid or not self.shared: - return uid - name = self.name(role) - if not self.owned(name, self.state["tokens"][role]): - raise UnknownOutcome(f"Cannot prove ownership of {name} on every replica") - uid = next(iter(self.catalog(name).values()))[0] - if role == "dual": - self.state["dual_uuid"] = uid - else: - self.state["helpers"][role] = uid - self.save() - return uid - - def owned(self, name, token): - """Every replica shows `name` with our token and one UUID; another owner's is never adopted.""" - catalog = self.catalog(name) - foreign = {host: row[2] for host, row in catalog.items() if row[2] != token} - if foreign: - raise UnknownOutcome(f"{name} carries another owner's comment: {foreign}") - return set(catalog) == self.hosts and len({row[0] for row in catalog.values()}) == 1 - - def proven(self, role): - """Our object is on every replica: by UUID once bound and, on Cloud, by its token too. - - `new` alone gives up its token after binding, for the replacement's own COMMENT. - """ - name = self.name(role) - uid = self.dual_uuid() if role == "dual" else self.state["helpers"][role] - token_checked = self.shared and not (role == "new" and uid) - if token_checked and not self.owned(name, self.state["tokens"][role]): - return False - return uid is None or self.converged(name, uid) - - def qualified(self, name): - return f"{_id(self.database)}.{_id(name)}" - - def system(self, table): - return ( - f"clusterAllReplicas({_lit(self.cluster)}, system.{table})" - if self.cluster - else f"system.{table}" - ) - - def rows(self, sql, parameters=None): - self.budget.check("rebuild inspection") - return self.client.query( - sql, parameters=parameters or {}, settings={"skip_unavailable_shards": 0} - ).result_rows - - def command(self, sql, settings=None): - self.budget.check("rebuild server command") - settings = dict(settings or {}) - if self.runtime.deployment.database_engine == "Replicated" and sql.startswith("CREATE "): - # Query-scoped documented opt-ins preserve our journaled UUID and - # isolated Keeper paths. Value 2 would silently replace them. - settings.update( - database_replicated_allow_explicit_uuid=1, - database_replicated_allow_replicated_engine_arguments=1, - ) - result = self.client.command(sql, settings=settings) - self.budget.check("rebuild server command completion") - return result - - def catalog(self, name): - rows = self.rows( - f"SELECT materialize(hostName()), toString(uuid), engine, comment FROM {self.system('tables')} " - "WHERE database = {db:String} AND name = {name:String}", - {"db": self.database, "name": name}, - ) - if len(rows) != len({str(row[0]) for row in rows}): - raise WaitingError(f"Duplicate catalog host for {name}") - return { - str(host): (str(uid), str(engine), str(comment)) for host, uid, engine, comment in rows - } - - def uuid_map(self, name, expected=None): - catalog = self.catalog(name) - if set(catalog) != self.hosts: - raise WaitingError(f"Incomplete catalog for {name}: {catalog}") - result = {host: values[0] for host, values in catalog.items()} - expected_map = ( - expected if isinstance(expected, dict) else {host: expected for host in self.hosts} - ) - if expected is not None and result != expected_map: - raise UnknownOutcome(f"Table {name}: expected UUID {expected}, observed {result}") - return result - - def converged(self, name, uid): - catalog = self.catalog(name) - expected = uid if isinstance(uid, dict) else {host: uid for host in self.hosts} - return ( - set(catalog) == self.hosts - and {host: row[0] for host, row in catalog.items()} == expected - ) - - def absent(self, name): - return not self.catalog(name) - - def time(self): - return str(self.rows("SELECT toString(now64(6))")[0][0]) - - def wait(self, label, predicate): - self.budget.check(label) - while not predicate(): - self.budget.pause(label) - self.budget.complete(label) - self.budget.check(label) - - def ddl(self, key, sql, converged): - """Never reissue a checkpointed statement after an ambiguous outcome.""" - operations = self.state.setdefault("ddl", {}) - intent = operations.get(key) - fresh = intent is None - if fresh: - if converged(): - raise UnknownOutcome(f"Unowned pre-existing DDL result for {key}") - intent = {"token": f"chm-rebuild-{uuid4()}", "acknowledged": False} - if self.ddl_cluster: - intent["receipt"] = self.waiter.prepare(self.ddl_cluster, intent["token"]) - operations[key] = intent - self.save() - self.command( - sql, - ( - {"log_comment": intent["token"], "distributed_ddl_task_timeout": 0} - if self.ddl_cluster - else {"log_comment": intent["token"]} - ), - ) - intent["acknowledged"] = True - self.save() - if self.ddl_cluster and not intent.get("done"): - self.waiter.wait(intent["receipt"], self.save) - if converged(): - intent["done"] = True - self.save() - return - if self.ddl_cluster: - self.wait(f"DDL {key} replica catalogs", converged) - elif fresh or intent["acknowledged"]: - self.wait(f"DDL {key} replica catalogs", converged) - else: - raise UnknownOutcome(f"Local DDL {key} has no acknowledgement or catalog proof") - intent["done"] = True - self.save() - - def partition_ids(self, name): - rows = self.rows( - f"SELECT DISTINCT partition_id FROM {self.system('parts')} " - "WHERE database = {db:String} AND table = {name:String} AND active AND rows > 0 ORDER BY partition_id", - {"db": self.database, "name": name}, - ) - return sorted({str(row[0]) for row in rows}) - - def counts(self, name, partition=None): - table = self.qualified(name) - source = f"clusterAllReplicas({_lit(self.cluster)}, {table})" if self.cluster else table - where = " WHERE _partition_id = {part:String}" if partition is not None else "" - rows = self.rows( - f"SELECT materialize(hostName()) AS host, count() FROM {source}{where} GROUP BY host", - {"part": partition} if partition is not None else {}, - ) - counts = {str(host): int(total) for host, total in rows} - if not set(counts).issubset(self.hosts): - raise WaitingError(f"Unexpected count hosts for {name}: {counts}") - return {host: counts.get(host, 0) for host in self.hosts} - - def equal_count(self, name, partition, expected): - return set(self.counts(name, partition).values()) == {expected} - - def local_count(self, name, partition): - return int( - self.rows( - f"SELECT count() FROM {self.qualified(name)} " - "WHERE _partition_id = {part:String}", - {"part": partition}, - )[0][0] - ) - - def _hosts(self): - hosts = [ - str(row[0]) - for row in self.rows(f"SELECT materialize(hostName()) FROM {self.system('one')}") - ] - if not hosts or len(hosts) != len(set(hosts)): - raise WaitingError(f"Cannot prove distinct rebuild replicas: {hosts}") - return set(hosts) - - -class _Lock: - def __init__(self, context): - self.ctx = context - self.name = f"_chm_rebuild_lock_{context.table}" - self.owner = str(uuid4()) - # Cloud assigns the UUID itself; the owner comment proves ownership until it is read back. - self.uuid = None if context.shared else str(uuid4()) - self.stop = threading.Event() - self.thread = None - self.failure = None - - def __enter__(self): - if not self.ctx.absent(self.name): - self._refuse() - try: - self.ctx.command( - self._create(), {"distributed_ddl_task_timeout": 0} if self.ctx.ddl_cluster else {} - ) - except Exception as exc: - if not self.ctx.absent(self.name): - self._refuse() - raise UnknownOutcome( - f"Lock CREATE failed or outcome uncertain; inspect replicas before retry: {exc}" - ) from exc - self.ctx.wait("lock ownership on every replica", self._owned) - self.check() - self._beat() - self.thread = threading.Thread( - target=self._heartbeat, name="chm-rebuild-heartbeat", daemon=True - ) - self.thread.start() - return self - - def __exit__(self, exc_type, exc, tb): - self.stop.set() - if self.thread is not None: - self.thread.join(timeout=_HEARTBEAT_SECONDS + 3) - if ( - exc_type is None - and self.failure is None - and self.thread is not None - and not self.thread.is_alive() - ): - self.check() - sql = f"DROP TABLE {self.ctx.qualified(self.name)}" - if self.ctx.ddl_cluster: - sql += f" ON CLUSTER {_id(self.ctx.ddl_cluster)}" - self.ctx.command(sql + " SYNC") - self.ctx.wait("owned lock release", lambda: self.ctx.absent(self.name)) - elif exc_type is None: - raise WaitingError( - f"Lock heartbeat failed or did not stop; lock retained: {self.failure}" - ) - return False - - def _create(self): - comment = _lit("chm-rebuild-owner:" + self.owner) - schema = "(owner String, heartbeat DateTime64(6) DEFAULT now64(6))" - if self.ctx.shared: - return ( - f"CREATE TABLE {self.ctx.qualified(self.name)} {schema} " - f"ENGINE = MergeTree ORDER BY heartbeat COMMENT {comment}" - ) - replicated = ( - len(self.ctx.hosts) > 1 or self.ctx.runtime.deployment.database_engine == "Replicated" - ) - path = f"/clickhouse/tables/{{shard}}/chm-rebuild-lock/{self.ctx.database}/{self.name}/{self.uuid}" - engine = ( - f"ReplicatedMergeTree({_lit(path)}, '{{replica}}')" if replicated else "MergeTree()" - ) - cluster = f" ON CLUSTER {_id(self.ctx.ddl_cluster)}" if self.ctx.ddl_cluster else "" - return ( - f"CREATE TABLE {self.ctx.qualified(self.name)} UUID {_lit(self.uuid)}{cluster} " - f"{schema} ENGINE = {engine} ORDER BY heartbeat COMMENT {comment}" - ) - - def _owned(self): - catalog = self.ctx.catalog(self.name) - if set(catalog) != self.ctx.hosts or any( - comment != f"chm-rebuild-owner:{self.owner}" for _, _, comment in catalog.values() - ): - return False - uids = {uid for uid, _, _ in catalog.values()} - if self.uuid is None and len(uids) == 1: - self.uuid = next(iter(uids)) - return uids == {self.uuid} - - def check(self): - if self.failure is not None or not self._owned(): - raise UnknownOutcome(f"Lock ownership lost or heartbeat stopped: {self.failure}") - - def _beat(self): - control = self.ctx.runtime.control_factory() - try: - control.command( - f"INSERT INTO {self.ctx.qualified(self.name)} (owner) " - f"VALUES ({_lit(self.owner)})", - settings={"async_insert": 0}, - ) - finally: - if hasattr(control, "close"): - control.close() - - def _heartbeat(self): - while not self.stop.wait(_HEARTBEAT_SECONDS): - try: - self._beat() - except Exception as exc: - self.failure = exc - return - - def _refuse(self): - catalog = self.ctx.catalog(self.name) - owners = {host: row[2] for host, row in catalog.items()} - try: - source = ( - f"clusterAllReplicas({_lit(self.ctx.cluster)}, {self.ctx.qualified(self.name)})" - if self.ctx.cluster - else self.ctx.qualified(self.name) - ) - rows = self.ctx.rows( - f"SELECT materialize(hostName()), max(heartbeat), " - f"dateDiff('second', max(heartbeat), now64(6)) " - f"FROM {source} GROUP BY hostName()" - ) - heartbeat = { - str(host): { - "last": str(last), - "age_seconds": int(age), - "stale": int(age) >= _STALE_SECONDS, - } - for host, last, age in rows - } - except Exception as exc: - heartbeat = {"unavailable": str(exc)} - raise WaitingError( - f"Rebuild lock {self.ctx.qualified(self.name)} exists; owners={owners}; " - f"heartbeat={heartbeat}. Expiry is diagnostic only; no automatic takeover. " - "Operator: establish prior runner cannot resume; reconcile in-flight copy/DDL queries, " - "waiting journal and all-host UUIDs; explicitly release lock on all replicas; rerun. " - "This invocation will not release the lock." - ) - - -def _run_locked(ctx, lock): - runtime = ctx.runtime - refreshed = runtime.refresh() - if refreshed is not None: - runtime.record.clear() - runtime.record.update(refreshed) - state = runtime.record.get("rebuild") - if state and state.get("result") is not None: - _verify_complete(ctx) - return state["result"] - if state and state.get("validation_rejection"): - raise DependentValidationError(state["validation_rejection"]) - if not state: - _preflight(ctx) - _initialize(ctx) - lock.check() - _resolve_exchange(ctx) - if not _swapped(ctx): - _helpers(ctx) - _validate_owned_dependents(ctx) - _dual(ctx) - lock.check() - _drain(ctx) - _snapshot(ctx) - _copy(ctx, lock) - if ctx.shared or ctx.definition.target.engine.startswith("Replicated"): - _sync_replica(ctx, ctx.name("new")) - _swap(ctx, lock) - _cleanup(ctx, lock) - _reload_owned_dictionaries(ctx) - _async_report(ctx) - _stranded(ctx) - result = _result(ctx) - ctx.state["result"] = result - ctx.save() - return result - - -def _validate_owned_dependents(ctx): - if ctx.state.get("dependents_validated"): - return - inventory = ctx.dependents or inspect_dependents(ctx.client, ctx.database, ctx.table) - validate_dependents(ctx.client, ctx.definition, inventory) - ctx.state["dependent_dictionaries"] = [item.name for item in inventory.dictionaries] - ctx.state["dependents_validated"] = True - ctx.save() - - -def _discard_validation(ctx, error): - state = ctx.runtime.record.get("rebuild") - if state is not None: - # A dual view may exist before its UUID is bound on Cloud: the DDL intent says so. - dual = state.get("dual_uuid") or state["ddl"].get("create_dual") - if dual or state["snapshot"].get("started") or state["parts"]: - raise UnknownOutcome( - f"{error}; dependent validation/reload failed after copying began; lock retained" - ) from error - ctx.uuid_map(ctx.table, state["old_uuids"]) - state["validation_rejection"] = str(error) - ctx.save() - for role in ("stage", "snap", "new"): - name = ctx.name(role) - if ctx.absent(name): - continue - ctx.uuid_map(name, ctx.bound_uuid(role)) - cluster = f" ON CLUSTER {_id(ctx.ddl_cluster)}" if ctx.ddl_cluster else "" - ctx.ddl( - f"reject_drop_{role}", - f"DROP TABLE {ctx.qualified(name)}{cluster} SYNC", - lambda target=name: ctx.absent(target), - ) - ctx.runtime.record.update(phase="rejected", repeat_safe=True) - ctx.save() - _progress("Dependent validation rejected before copying; owned helpers removed") - - -def _reload_owned_dictionaries(ctx): - if ctx.state.get("dictionaries_reloaded"): - return - names = tuple(ctx.state.get("dependent_dictionaries", ())) - ctx.budget.check("dependent dictionary reload") - reload_dictionaries(ctx.client, DictionaryReload(ctx.database, names, ctx.cluster)) - ctx.budget.check("dependent dictionary reload completion") - ctx.state["dictionaries_reloaded"] = True - ctx.save() - - -def _preflight(ctx): - definition = ctx.definition - if len(ctx.hosts) > 1 and not definition.source.engine.startswith(("Replicated", "Shared")): - raise WaitingError( - "Multi-host unreplicated source cannot use single-node physical partition copy" - ) - request = RebuildRequest( - definition.database, - definition.source, - definition.target, - ctx.cluster, - definition.allow_unacknowledged_async_loss, - ((definition.source, definition.source), (definition.target, definition.target)), - definition.writers_on_this_service_only, - ) - findings = inspect_rebuild(ctx.client, request).findings - for finding in findings: - if finding.severity == "warning": - _progress(f"Warning [{finding.code}]: {finding.message} {finding.details}") - refusals = [ - f"{finding.code}: {finding.message} {finding.details}" - for finding in findings - if finding.severity == "refusal" - ] - if refusals: - raise WaitingError("Rebuild read-only preflight refused: " + "; ".join(refusals)) - ctx.dependents = inspect_dependents(ctx.client, ctx.database, ctx.table) - - -def _initialize(ctx): - record = ctx.runtime.record - state = record.get("rebuild") - if state and state.get("helpers"): - if not record.get("source_ddl"): - raise UnknownOutcome("Original source_ddl missing; cannot safely resume post-swap") - return - old = ctx.uuid_map(ctx.table) - names = [ctx.name(role) for role in ("new", "snap", "stage", "dual")] - rollback = f"{ctx.table}__chm_old_{ctx.definition.revision}" - for name in names + [rollback]: - if not ctx.absent(name): - raise WaitingError(f"Unowned helper exists: {ctx.qualified(name)}") - state = record.setdefault("rebuild", {}) - roles = ("new", "snap", "stage") - state.update( - { - "helpers": {role: None if ctx.shared else str(uuid4()) for role in roles}, - "old_uuids": old, - "rollback_table": rollback, - "snapshot": {}, - "parts": {}, - "ddl": {}, - } - ) - if ctx.shared: - # Journaled before any CREATE: the token is this run's only proof of ownership. - revision = ctx.definition.revision - state["tokens"] = { - role: f"chm-rebuild:{revision}:{role}:{uuid4().hex}" for role in (*roles, "dual") - } - state["dual_uuid"] = None - ctx.save() - - -def _helpers(ctx): - for role in ("new", "snap", "stage"): - ctx.ddl( - f"create_{role}", - helper_ddl(ctx.definition, role, ctx.identity(role)), - lambda r=role: ctx.proven(r), - ) - ctx.bound_uuid(role) - if ctx.shared: - _restore_new_comment(ctx) - - -def _restore_new_comment(ctx): - """Once bound by UUID, `new` drops its owner token for the replacement's own COMMENT.""" - sql, comment = target_comment(ctx.definition) - name, uid = ctx.name("new"), ctx.state["helpers"]["new"] - - def restored(): - catalog = ctx.catalog(name) - return set(catalog) == ctx.hosts and all( - row[0] == uid and row[2] == comment for row in catalog.values() - ) - - ctx.ddl("comment_new", sql, restored) - - -def _dual(ctx): - state = ctx.state - name = ctx.name("dual") - if "dual_start" not in state: - state["dual_start"] = ctx.time() - ctx.save() - if "dual_uuid" not in state: - if not ctx.absent(name): - raise UnknownOutcome(f"Unowned dual view exists: {ctx.catalog(name)}") - state["dual_uuid"] = str(uuid4()) - ctx.save() - ctx.ddl( - "create_dual", dual_ddl(ctx.definition, ctx.identity("dual")), lambda: ctx.proven("dual") - ) - uid = ctx.bound_uuid("dual") - ctx.wait("dual visible on every replica", lambda: ctx.converged(name, uid)) - if "barrier" not in state: - if ctx.shared: - _grace(ctx) - state["barrier"] = ctx.time() - ctx.save() - - -def _grace(ctx): - """Let every replica apply the view's metadata before inserts are drained (Cloud). - - The session is queried throughout: Cloud closes a connection idle for 10 s, and the - reconnect may reach a replica that does not hold this session. - """ - label = f"{ctx.definition.grace_seconds:g}s grace after the copying view" - deadline = time.monotonic() + ctx.definition.grace_seconds - while (remaining := deadline - time.monotonic()) > 0: - ctx.rows("SELECT 1") - ctx.budget.check(label) - time.sleep(min(remaining, 1.0)) - - -def _drain(ctx): - if ctx.state.get("drained"): - return - _drain_before(ctx, ctx.state["barrier"]) - if ctx.shared or ctx.definition.source.engine.startswith("Replicated"): - _sync_source(ctx) - ctx.state["drained"] = True - ctx.save() - - -def _drain_before(ctx, barrier): - sql = ( - "SELECT materialize(hostName()), query_id FROM " - + ctx.system("processes") - + " WHERE query_kind IN ('Insert', 'AsyncInsertFlush') " - "AND now64(6) - toIntervalMillisecond(toUInt64(elapsed * 1000)) " - "<= toDateTime64({barrier:String}, 6)" - ) - ctx.wait("pre-barrier insert pipelines", lambda: not ctx.rows(sql, {"barrier": barrier})) - - -def _sync_source(ctx): - _sync_replica(ctx, ctx.table) - - -def _sync_replica(ctx, name): - cluster = f" ON CLUSTER {_id(ctx.cluster)}" if ctx.cluster else "" - # LIGHTWEIGHT drains the captured fetch queue without requiring future - # inserts or background merges to stop. - ctx.command(f"SYSTEM SYNC REPLICA{cluster} {ctx.qualified(name)} LIGHTWEIGHT") - - -def _snapshot(ctx): - snapshot = ctx.state["snapshot"] - if snapshot.get("complete"): - return - if snapshot.get("started"): - if any(part.get("state") == "moved" for part in ctx.state["parts"].values()): - raise UnknownOutcome("Incomplete snapshot after moved partition; reconcile manually") - for partition in ctx.partition_ids(ctx.name("snap")): - _drop_partition(ctx, ctx.name("snap"), partition) - snapshot.clear() - ctx.save() - # snap is plain MergeTree without TTL, so its merges keep every counted row. - snapshot.update({"partitions": ctx.partition_ids(ctx.table), "started": True, "counts": {}}) - ctx.save() - for partition in snapshot["partitions"]: - ctx.command( - f"ALTER TABLE {ctx.qualified(ctx.name('snap'))} " - f"ATTACH PARTITION ID {_lit(partition)} FROM {ctx.qualified(ctx.table)}" - ) - expected = ctx.local_count(ctx.name("snap"), partition) - ctx.wait( - f"snapshot {partition} replicated", - lambda p=partition, n=expected: ctx.equal_count(ctx.name("snap"), p, n), - ) - snapshot["counts"][partition] = expected - ctx.save() - snapshot["complete"] = True - snapshot["end"] = ctx.time() - ctx.save() - - -def _copy(ctx, lock): - for partition, expected in sorted(ctx.state["snapshot"]["counts"].items()): - lock.check() - _copy_partition(ctx, partition, expected) - - -def _copy_partition(ctx, partition, expected): - item = ctx.state["parts"].setdefault(partition, {}) - if item.get("state") == "moved": - _progress(f"Skipping previously moved partition {partition}") - return - stage = ctx.name("stage") - if expected == 0: - if not ctx.equal_count(stage, partition, 0): - raise UnknownOutcome(f"Nonempty stage for empty snapshot partition {partition}") - item["state"] = "moved" - ctx.save() - _progress(f"Skipping empty snapshot partition {partition}") - return - other = set(ctx.partition_ids(stage)) - {partition} - if other: - raise UnknownOutcome( - f"Stage has unexpected partition IDs {sorted(other)}; projection may change partition key" - ) - counts = ctx.counts(stage, partition) - if item.get("state") == "staged" and item.get("move_intent"): - observed = set(counts.values()) - if not observed.issubset({0, expected}): - raise UnknownOutcome(f"Partial MOVE for {partition}: {counts}; reconcile before retry") - if observed == {0, expected}: - ctx.wait( - f"prior MOVE {partition} replica convergence", - lambda: ctx.equal_count(stage, partition, 0), - ) - counts = ctx.counts(stage, partition) - if item.get("state") == "staged" and set(counts.values()) == {0}: - _progress(f"Recovering MOVE for staged partition {partition}") - item["state"] = "moved" - ctx.save() - return - if item.get("state") != "staged" or set(counts.values()) != {expected}: - _progress(f"Recovering/copying partition {partition}") - _kill_orphan(ctx, _copy_id(ctx, partition)) - _drop_partition(ctx, stage, partition) - item["state"] = "copying" - ctx.save() - columns = ", ".join(_id(name) for name in ctx.definition.columns) - sql = ( - f"INSERT INTO {ctx.qualified(stage)} ({columns}) " - f"SELECT {ctx.definition.projection} FROM {ctx.qualified(ctx.name('snap'))} " - f"WHERE _partition_id = {_lit(partition)}" - ) - ctx.command( - sql, - { - "query_id": _copy_id(ctx, partition), - "max_threads": 2, - "max_insert_threads": 1, - "max_memory_usage": _MAX_COPY_MEMORY, - "max_block_size": 1024, - "async_insert": 0, - }, - ) - _verify_stage(ctx, partition, expected) - item.update({"state": "staged", "rows": expected}) - ctx.save() - _verify_stage(ctx, partition, expected) - item["move_intent"] = True - ctx.save() - ctx.command( - f"ALTER TABLE {ctx.qualified(stage)} MOVE PARTITION ID {_lit(partition)} " - f"TO TABLE {ctx.qualified(ctx.name('new'))}" - ) - ctx.wait(f"stage partition {partition} moved", lambda: ctx.equal_count(stage, partition, 0)) - item["state"] = "moved" - ctx.save() - _progress(f"Moved partition {partition}: {expected} snapshot rows") - - -def _verify_stage(ctx, partition, expected): - def complete(): - ids = set(ctx.partition_ids(ctx.name("stage"))) - if ids - {partition}: - raise UnknownOutcome( - f"Projection changed partition ID for {partition}: stage has {sorted(ids)}" - ) - return ctx.equal_count(ctx.name("stage"), partition, expected) - - ctx.wait(f"stage {partition} count and replica convergence", complete) - - -def _kill_orphan(ctx, query_id): - sql = "KILL QUERY" - if ctx.cluster: - sql += f" ON CLUSTER {_id(ctx.cluster)}" - ctx.command(sql + f" WHERE query_id = {_lit(query_id)} SYNC") - ctx.wait( - f"orphan copy {query_id} stopped", - lambda: not ctx.rows( - f"SELECT materialize(hostName()) FROM {ctx.system('processes')} " - "WHERE query_id = {qid:String}", - {"qid": query_id}, - ), - ) - - -def _drop_partition(ctx, table, partition): - ctx.command(f"ALTER TABLE {ctx.qualified(table)} DROP PARTITION ID {_lit(partition)}") - ctx.wait( - f"drop {table} partition {partition} propagated", - lambda: ctx.equal_count(table, partition, 0), - ) - - -def _copy_id(ctx, partition): - return f"chm-rebuild-{ctx.definition.revision}-{ctx.database}.{ctx.table}-{partition}" - - -def _exchange_evidence(ctx): - old = ctx.state["old_uuids"] - new = ctx.state["helpers"]["new"] - catalog = ctx.catalog(ctx.table) - if set(catalog) != ctx.hosts: - raise UnknownOutcome(f"Source catalog incomplete: {catalog}") - observed = {host: row[0] for host, row in catalog.items()} - if any(uid not in {new, old.get(host)} for host, uid in observed.items()): - raise UnknownOutcome(f"Foreign source UUIDs during rebuild: {observed}") - if set(observed.values()) == {new}: - return "swapped" - if observed == old: - return "old" - return "mixed" - - -def _swapped(ctx): - outcome = _exchange_evidence(ctx) - if outcome == "mixed": - raise UnknownOutcome(f"Mixed exchange UUIDs: {ctx.catalog(ctx.table)}") - return outcome == "swapped" - - -def _resolve_exchange(ctx): - intent = ctx.state.get("ddl", {}).get("exchange") - if intent is None: - if _exchange_evidence(ctx) == "mixed": - raise UnknownOutcome("Mixed UUIDs with no owned EXCHANGE receipt") - return - if _exchange_evidence(ctx) == "swapped": - return - old = ctx.state["old_uuids"] - new = ctx.state["helpers"]["new"] - if ctx.ddl_cluster: - ctx.waiter.wait(intent["receipt"], ctx.save) - ctx.wait( - "EXCHANGE catalog convergence", - lambda: ctx.converged(ctx.table, new) and ctx.converged(ctx.name("new"), old), - ) - elif intent.get("acknowledged"): - ctx.wait( - "EXCHANGE catalog convergence", - lambda: ctx.converged(ctx.table, new) and ctx.converged(ctx.name("new"), old), - ) - else: - raise UnknownOutcome("Unacknowledged EXCHANGE cannot be retried without catalog proof") - intent["done"] = True - ctx.save() - - -def _swap(ctx, lock): - state = ctx.state - old = state["old_uuids"] - new = state["helpers"]["new"] - if "swap_intent" not in state: - state["swap_intent"] = ctx.time() - ctx.save() - lock.check() - if not _swapped(ctx): - if not ctx.shared: - _flush_async(ctx) - sql = f"EXCHANGE TABLES {ctx.qualified(ctx.table)} AND {ctx.qualified(ctx.name('new'))}" - if ctx.ddl_cluster: - sql += f" ON CLUSTER {_id(ctx.ddl_cluster)}" - ctx.ddl( - "exchange", - sql, - lambda: ctx.converged(ctx.table, new) and ctx.converged(ctx.name("new"), old), - ) - ctx.wait( - "new table UUID on every replica", - lambda: ctx.converged(ctx.table, new) and ctx.converged(ctx.name("new"), old), - ) - state["exchanged"] = True - state.setdefault("exchanged_at", ctx.time()) - ctx.save() - - -def _cleanup(ctx, lock): - state = ctx.state - if not _swapped(ctx): - raise UnknownOutcome("Cleanup refused without all-host new UUID") - old = state["old_uuids"] - lock.check() - dual = ctx.name("dual") - sql = f"DROP TABLE {ctx.qualified(dual)}" - if ctx.ddl_cluster: - sql += f" ON CLUSTER {_id(ctx.ddl_cluster)}" - ctx.ddl("drop_dual", sql + " SYNC", lambda: ctx.absent(dual)) - ctx.wait("dual DROP SYNC on every replica", lambda: ctx.absent(dual)) - # Catalog absence alone does not prove an unacknowledged DROP SYNC finished - # draining pipelines. Re-establish that barrier across every replica. - if not state.get("dual_drained"): - if "dual_drop_barrier" not in state: - state["dual_drop_barrier"] = ctx.time() - ctx.save() - _drain_before(ctx, state["dual_drop_barrier"]) - state["dual_drained"] = True - ctx.save() - rollback = state["rollback_table"] - if not ctx.converged(rollback, old): - ctx.uuid_map(ctx.name("new"), old) - sql = f"RENAME TABLE {ctx.qualified(ctx.name('new'))} TO {ctx.qualified(rollback)}" - if ctx.ddl_cluster: - sql += f" ON CLUSTER {_id(ctx.ddl_cluster)}" - ctx.ddl( - "rename_old", sql, lambda: ctx.converged(rollback, old) and ctx.absent(ctx.name("new")) - ) - ctx.uuid_map(rollback, old) - for role in ("snap", "stage"): - name = ctx.name(role) - sql = f"DROP TABLE {ctx.qualified(name)}" - if ctx.ddl_cluster: - sql += f" ON CLUSTER {_id(ctx.ddl_cluster)}" - ctx.ddl(f"drop_{role}", sql + " SYNC", lambda n=name: ctx.absent(n)) - state["cleanup"] = True - ctx.save() - - -def _async_report(ctx): - state = ctx.state - if "async_flush_errors" in state: - return - if not state.get("swap_intent"): - raise UnknownOutcome("Swap timestamp missing; cannot attribute async flush errors") - if "report_end" not in state: - if ctx.shared: - # Cloud forbids the flush; EXCHANGE itself drains the old queue (Cloud probes). - _await_old_queue(ctx, state.get("exchanged_at") or ctx.time()) - else: - # Queues follow the old storage UUID. Flushing the now-live name would - # flush the replacement and miss the old queue's later error 741. - _flush_async(ctx, state["rollback_table"]) - _drain_before(ctx, ctx.time()) - ctx.command(_flush_logs_sql(ctx)) - state["report_end"] = ctx.time() - ctx.save() - _require_async_log(ctx) - rows = _settled_flush_errors(ctx) - state["async_flush_errors"] = { - "count": len(rows), - "bytes": sum(int(row[2]) for row in rows), - "query_ids": sorted({str(row[1]) for row in rows}), - # Cloud's report is a lower bound: no flush, and logs of other services are unseen. - "complete": not ctx.shared, - "replicas": sorted(ctx.hosts), - } - ctx.save() - _progress(f"Old-UUID asynchronous FlushErrors: {state['async_flush_errors']}") - - -def _await_old_queue(ctx, cutoff): - sql = ( - "SELECT count() FROM " - + ctx.system("asynchronous_inserts") - + " WHERE database = {db:String} AND table = {table:String} " - "AND first_update <= toDateTime64({cutoff:String}, 6)" - ) - parameters = {"db": ctx.database, "table": ctx.table, "cutoff": cutoff} - ctx.wait("pre-swap async insert queue", lambda: not ctx.rows(sql, parameters)[0][0]) - - -def _settled_flush_errors(ctx): - """Old-UUID FlushErrors; on Cloud, read until two consecutive reads agree.""" - ctx.command(_flush_logs_sql(ctx)) - rows = _flush_errors(ctx) - while ctx.shared: - ctx.budget.pause("asynchronous insert log settling") - ctx.command(_flush_logs_sql(ctx)) - again = _flush_errors(ctx) - if again == rows: - break - rows = again - return rows - - -def _flush_errors(ctx): - state = ctx.state - return sorted( - ctx.rows( - f"SELECT materialize(hostName()), query_id, bytes FROM {ctx.system('asynchronous_insert_log')} " - "WHERE database = {db:String} AND table = {table:String} AND status = 'FlushError' " - "AND arrayExists(uuid -> position(exception, uuid) > 0, {old:Array(String)}) " - "AND flush_time_microseconds >= toDateTime64({start:String}, 6) " - "AND flush_time_microseconds <= toDateTime64({end:String}, 6)", - { - "db": ctx.database, - "table": ctx.table, - "old": sorted(set(state["old_uuids"].values())), - "start": state["swap_intent"], - "end": state["report_end"], - }, - ) - ) - - -def _flush_logs_sql(ctx): - return "SYSTEM FLUSH LOGS" + (f" ON CLUSTER {_id(ctx.cluster)}" if ctx.cluster else "") - - -def _require_async_log(ctx): - columns = { - str(row[0]) - for row in ctx.rows( - "SELECT name FROM system.columns WHERE database = 'system' AND table = 'asynchronous_insert_log'" - ) - } - required = { - "database", - "table", - "status", - "bytes", - "query_id", - "exception", - "flush_time_microseconds", - } - if not required.issubset(columns): - raise WaitingError(f"Async log lacks old-UUID evidence: {sorted(required - columns)}") - - -def _flush_async(ctx, table=None): - cluster = f" ON CLUSTER {_id(ctx.cluster)}" if ctx.cluster else "" - ctx.command(f"SYSTEM FLUSH ASYNC INSERT QUEUE{cluster} {ctx.qualified(table or ctx.table)}") - - -def _stranded(ctx): - """Find inserts that wrote only the old storage at the swap, and stop for the operator. - - An insert that resolves the table before EXCHANGE and builds its view chain after it - skips the copying view; its rows exist only in the rollback table. - """ - state = ctx.state - if "stranded" in state: - if state["stranded"]["rows"] and not state["stranded"]["acknowledged"]: - # A stop retains the lock, so only the operator's release lets a run get here. - state["stranded"]["acknowledged"] = True - ctx.save() - return - ctx.command(_flush_logs_sql(ctx)) - rows = _stranded_parts(ctx) - state["stranded"] = { - "rows": sum(int(row[1]) for row in rows), - "query_ids": sorted({str(row[0]) for row in rows}), - "acknowledged": False, - } - ctx.save() - if state["stranded"]["rows"]: - raise StrandedRows( - f"{state['stranded']['rows']} acknowledged rows from inserts " - f"{state['stranded']['query_ids']} reached only " - f"{ctx.qualified(state['rollback_table'])}. Copy them into " - f"{ctx.qualified(ctx.table)}, then release the rebuild lock and run up again." - ) - - -def _stranded_parts(ctx): - """New old-storage parts since the swap from queries that wrote nothing to the new table.""" - state = ctx.state - return ctx.rows( - f"SELECT query_id, sum(rows) FROM {ctx.system('part_log')} " - "WHERE event_type = 'NewPart' AND has({old:Array(String)}, toString(table_uuid)) " - "AND event_time_microseconds >= toDateTime64({start:String}, 6) " - f"AND query_id NOT IN (SELECT query_id FROM {ctx.system('part_log')} " - "WHERE event_type = 'NewPart' AND has({new:Array(String)}, toString(table_uuid))) " - "GROUP BY query_id", - { - "old": _uuids(state["old_uuids"]), - "new": _uuids(state["helpers"]["new"]), - "start": state["swap_intent"], - }, - ) - - -def _uuids(value): - return sorted(set(value.values()) if isinstance(value, dict) else {value}) - - -def _result(ctx): - state = ctx.state - old = state["old_uuids"] - return { - "old_uuids": ctx.uuid_map(state["rollback_table"], old), - "new_uuids": ctx.uuid_map(ctx.table, state["helpers"]["new"]), - "rollback_table": state["rollback_table"], - "duplicate_window": {"start": state["dual_start"], "end": state["snapshot"]["end"]}, - "async_flush_errors": state["async_flush_errors"], - "stranded": state["stranded"], - } - - -def _completed_without_lock(ctx): - if not ctx.absent(f"_chm_rebuild_lock_{ctx.table}"): - raise WaitingError( - "Completed rebuild has a retained lock; operator reconciliation/release required" - ) - _verify_complete(ctx) - - -def _verify_complete(ctx): - state = ctx.state - ctx.uuid_map(ctx.table, state["helpers"]["new"]) - ctx.uuid_map(state["rollback_table"], state["old_uuids"]) - for role in ("dual", "snap", "stage"): - if not ctx.absent(ctx.name(role)): - raise UnknownOutcome(f"Completed rebuild has surviving {role} helper") - - -def _progress(message): - print(f"Rebuild: {message}", file=sys.stderr, flush=True) - - -def _id(value): - return "`" + value.replace("`", "``") + "`" - - -def _lit(value): - return "'" + value.replace("\\", "\\\\").replace("'", "\\'") + "'" diff --git a/ch_migrate/rebuild_preflight.py b/ch_migrate/rebuild_preflight.py deleted file mode 100644 index a1929e3..0000000 --- a/ch_migrate/rebuild_preflight.py +++ /dev/null @@ -1,704 +0,0 @@ -"""Read-only, fail-closed assessment of a proposed table rebuild.""" - -from __future__ import annotations - -import re -from dataclasses import dataclass, field -from typing import Any, Literal - -from ch_migrate.classify import _tokens -from ch_migrate.introspect import TableDefinition - -_WINDOW_MINUTES = 60 -_LOG_LIMIT = 10001 -_SETTING_NAMES = ("async_insert", "wait_for_async_insert") -_REPLACING = {"ReplacingMergeTree", "ReplicatedReplacingMergeTree", "SharedReplacingMergeTree"} - - -@dataclass(frozen=True) -class RebuildRequest: - database: str - source: TableDefinition - target: TableDefinition - cluster: str | None = None - allow_unacknowledged_async_loss: bool = False - transfer_pairs: tuple[tuple[TableDefinition, TableDefinition], ...] = () - writers_on_this_service_only: bool = False - - -@dataclass(frozen=True) -class PreflightFinding: - code: str - severity: Literal["refusal", "warning"] - message: str - details: dict[str, Any] = field(default_factory=dict) - - -@dataclass(frozen=True) -class RebuildAssessment: - findings: tuple[PreflightFinding, ...] - bytes_on_disk: int - part_count: int - partition_parts: dict[str, int] - insert_rows_per_second: float - engine_note: str - - -def inspect_rebuild(client: Any, request: RebuildRequest) -> RebuildAssessment: - """Inspect live state; inspection errors propagate instead of implying safety.""" - scope = _Scope(client, request.cluster) - findings = _cloud_findings(request) + _definition_findings(request) - findings.extend(_topology_findings(scope, request)) - findings.extend(_replica_findings(scope, request)) - findings.extend(_mutation_findings(scope, request)) - storage = _parts(scope, request) - findings.extend(_capacity_findings(scope, request, storage)) - rate, writers = _writer_log(scope, request) - findings.extend(_writer_findings(scope, request, writers)) - findings.extend(_dedup_findings(scope, request)) - findings.extend(_part_log_findings(scope)) - return RebuildAssessment( - tuple(findings), - storage.bytes_on_disk, - storage.part_count, - storage.partition_parts, - rate, - _engine_note(request), - ) - - -@dataclass(frozen=True) -class _Storage: - bytes_on_disk: int - part_count: int - partition_parts: dict[str, int] - host_partitions: dict[tuple[str, str], int] - host_disks: dict[tuple[str, str], int] - - -@dataclass(frozen=True) -class _Scope: - client: Any - cluster: str | None - - def table(self, table: str) -> str: - if self.cluster: - return f"clusterAllReplicas({_literal(self.cluster)}, system.{table})" - return f"system.{table}" - - def log(self) -> str: - merged = "merge('system', '^query_log(_[0-9]+)?$')" - if self.cluster: - return f"clusterAllReplicas({_literal(self.cluster)}, {merged})" - return merged - - def rows(self, sql: str) -> list[tuple[Any, ...]]: - return self.client.query(sql).result_rows - - -def _cloud_findings(request: RebuildRequest) -> list[PreflightFinding]: - # Shared* engines exist only in ClickHouse Cloud's Shared databases. - if not request.source.engine.startswith("Shared"): - return [] - if not request.writers_on_this_service_only: - return [ - PreflightFinding( - "cloud_writers_undeclared", - "refusal", - "On ClickHouse Cloud, writers on another compute service are invisible to this " - "one, and an insert of theirs could be missing from the rebuilt table. Once every " - "writer to this table uses this service, declare writers_on_this_service_only=True " - "in the migration.", - {"engine": request.source.engine}, - ) - ] - return [ - PreflightFinding( - "cloud_writers_declared", - "warning", - "Declared that every writer uses this service; ch-migrate cannot verify that.", - {"engine": request.source.engine}, - ) - ] - - -def _part_log_findings(scope: _Scope) -> list[PreflightFinding]: - """After the swap, part_log is the only evidence of rows stranded in the old table.""" - hosts = scope.rows( - "SELECT materialize(hostName()) AS host, countIf(database = 'system' AND name = 'part_log') " - f"FROM {scope.table('tables')} GROUP BY host" - ) - missing = sorted(str(host) for host, present in hosts if not present) - if not missing: - return [] - return [ - PreflightFinding( - "part_log_unavailable", - "refusal", - "system.part_log is missing; the rebuild could not detect rows that reach only " - "the old table at the swap. Enable part_log, or run SYSTEM FLUSH LOGS if it is " - "configured but not yet created.", - {"hosts": missing}, - ) - ] - - -def _definition_findings(request: RebuildRequest) -> list[PreflightFinding]: - findings = [] - if request.source.engine.split("(", 1)[0] == "Distributed": - findings.append( - PreflightFinding( - "distributed_source", - "refusal", - "A Distributed source cannot be rebuilt atomically.", - ) - ) - if _normalized(request.source.partition_by or "tuple()") != _normalized( - request.target.partition_by or "tuple()" - ): - findings.append( - PreflightFinding( - "partition_key_change", - "refusal", - "rebuild_table cannot change the partition key.", - ) - ) - for source, destination in request.transfer_pairs: - differences = _transfer_differences(source, destination) - if differences: - findings.append( - PreflightFinding( - "transfer_structure_mismatch", - "refusal", - "Physical partition transfer requires matching structures and engines.", - { - "source": source.name, - "destination": destination.name, - "differences": differences, - }, - ) - ) - return findings - - -def _transfer_differences(source: TableDefinition, destination: TableDefinition) -> list[str]: - differences = [] - # Snapshot and stage helpers are deliberately plain MergeTree; partition transfer - # between MergeTree families works, but never across replication families. - if _replication_family(source.engine) != _replication_family(destination.engine): - differences.append("engine") - if _normalized(" ".join(source.order_by)) != _normalized(" ".join(destination.order_by)): - differences.append("sorting_key") - if _normalized(source.partition_by or "tuple()") != _normalized( - destination.partition_by or "tuple()" - ): - differences.append("partition_key") - - def physical(table: TableDefinition) -> list[tuple[str, str]]: - return [ - (col.name, _normalized(col.type)) - for col in table.columns - if col.default_kind != "ALIAS" - ] - - if physical(source) != physical(destination): - differences.append("physical_columns") - return differences - - -def _replication_family(engine: str) -> str: - match = re.match(r"(Replicated|Shared)?", engine.strip(), re.I) - return match.group(0).lower() - - -def _topology_findings(scope: _Scope, request: RebuildRequest) -> list[PreflightFinding]: - findings = [] - if request.cluster: - shards = scope.rows( - f"SELECT uniqExact(shard_num) FROM system.clusters WHERE cluster = {_literal(request.cluster)}" - ) - if not shards or not shards[0][0]: - raise RuntimeError(f"Configured ClickHouse cluster {request.cluster!r} is not visible") - if shards[0][0] > 1: - findings.append( - PreflightFinding( - "sharded_cluster", - "refusal", - "Rebuild cannot swap atomically across shards.", - {"shards": int(shards[0][0])}, - ) - ) - sql = ( - "SELECT materialize(hostName()), database, name, engine_full FROM " - + scope.table("tables") - + " WHERE engine = 'Distributed'" - ) - for host, database, name, engine in scope.rows(sql): - route = re.match( - r"^Distributed\s*\(\s*(['`\"]?)[^,]+?\1\s*,\s*(['`\"]?)([^,'`\"\s)]+)\2\s*,\s*(['`\"]?)([^,'`\"\s)]+)\4", - engine, - re.I, - ) - if route and route.group(3) == request.database and route.group(5) == request.source.name: - findings.append( - PreflightFinding( - "distributed_route", - "refusal", - "A Distributed table routes inserts to this source.", - {"host": str(host), "table": f"{database}.{name}"}, - ) - ) - elif request.source.name in engine and not route: - findings.append( - PreflightFinding( - "distributed_route_unknown", - "refusal", - "Cannot exclude a Distributed route to this source.", - {"host": str(host), "table": f"{database}.{name}"}, - ) - ) - return findings - - -def _replica_findings(scope: _Scope, request: RebuildRequest) -> list[PreflightFinding]: - if not request.source.engine.startswith("Replicated"): - return [] - rows = scope.rows( - "SELECT materialize(hostName()), zookeeper_path, replica_name, total_replicas FROM " - + scope.table("replicas") - + f" WHERE database = {_literal(request.database)} AND table = {_literal(request.source.name)}" - ) - if not rows: - raise RuntimeError("Cannot inspect replicated source membership") - if len({row[1] for row in rows}) != 1: - return [ - PreflightFinding( - "sharded_replica_groups", - "refusal", - "Source hosts belong to different replication groups.", - ) - ] - expected = max(int(row[3]) for row in rows) - if len(rows) != expected or len({row[2] for row in rows}) != len(rows): - return [ - PreflightFinding( - "incomplete_replica_scope", - "refusal", - "Configure a cluster covering every source replica before rebuilding.", - {"observed_replicas": len(rows), "expected_replicas": expected}, - ) - ] - return [] - - -def _mutation_findings(scope: _Scope, request: RebuildRequest) -> list[PreflightFinding]: - sql = ( - "SELECT materialize(hostName()), mutation_id FROM " - + scope.table("mutations") - + f" WHERE database = {_literal(request.database)} AND table = {_literal(request.source.name)} AND is_done = 0" - ) - rows = scope.rows(sql) - if not rows: - return [] - return [ - PreflightFinding( - "unfinished_mutations", - "refusal", - "Source has unfinished mutations on at least one replica.", - {"mutations": [{"host": str(host), "id": str(mid)} for host, mid in rows]}, - ) - ] - - -def _parts(scope: _Scope, request: RebuildRequest) -> _Storage: - sql = ( - "SELECT materialize(hostName()) AS host, partition_id, disk_name, count(), sum(bytes_on_disk) FROM " - + scope.table("parts") - + f" WHERE database = {_literal(request.database)} AND table = {_literal(request.source.name)}" - + " AND active = 1 GROUP BY host, partition_id, disk_name" - ) - partitions: dict[tuple[str, str], int] = {} - disks: dict[tuple[str, str], int] = {} - sizes: dict[str, int] = {} - counts: dict[str, int] = {} - for host, partition, disk, count, size in scope.rows(sql): - partitions[host, partition] = partitions.get((host, partition), 0) + int(count) - disks[host, disk] = disks.get((host, disk), 0) + int(size) - sizes[host] = sizes.get(host, 0) + int(size) - counts[host] = counts.get(host, 0) + int(count) - maxima: dict[str, int] = {} - for (_, partition), count in partitions.items(): - maxima[partition] = max(maxima.get(partition, 0), count) - return _Storage( - max(sizes.values(), default=0), max(counts.values(), default=0), maxima, partitions, disks - ) - - -def _capacity_findings( - scope: _Scope, request: RebuildRequest, storage: _Storage -) -> list[PreflightFinding]: - findings = [] - rows = scope.rows( - "SELECT materialize(hostName()), name, unreserved_space FROM " + scope.table("disks") - ) - disks = {(host, disk): int(free) for host, disk, free in rows} - for (host, disk), size in storage.host_disks.items(): - if (host, disk) not in disks: - raise RuntimeError(f"No capacity data for source disk {host}/{disk}") - if disks[host, disk] < 3 * size: - findings.append( - PreflightFinding( - "low_disk_space", - "warning", - "Less than approximately three times source bytes are free on a source disk.", - { - "host": host, - "disk": disk, - "available_bytes": disks[host, disk], - "source_bytes": size, - }, - ) - ) - configured = request.source.settings.get("parts_to_throw_insert") - thresholds = dict( - scope.rows( - "SELECT materialize(hostName()), value FROM " - + scope.table("merge_tree_settings") - + " WHERE name = 'parts_to_throw_insert'" - ) - ) - for (host, partition), count in storage.host_partitions.items(): - if configured is None and host not in thresholds: - raise RuntimeError(f"Cannot determine parts_to_throw_insert on {host}") - threshold = int(configured if configured is not None else thresholds[host]) - if threshold > 0 and count >= threshold * 0.8: - findings.append( - PreflightFinding( - "many_parts", - "warning", - "Partition approaches parts_to_throw_insert.", - { - "host": host, - "partition_id": partition, - "parts": count, - "threshold": threshold, - }, - ) - ) - return findings - - -def _writer_log(scope: _Scope, request: RebuildRequest) -> tuple[float, list[dict[str, Any]]]: - """Writers of the source table in the last hour, one per host, user and async settings. - - Queued inserts whose settings the log cannot prove are taken from the queue instead - and kept out of the logged groups. - """ - queued = _queued_writers(scope, request) - writers = _logged_writers(scope, request, {query_id for _, query_id in queued}) - writers.extend(queued.values()) - return sum(writer["rows"] for writer in writers) / (_WINDOW_MINUTES * 60), writers - - -def _logged_writers( - scope: _Scope, request: RebuildRequest, excluded: set[str] -) -> list[dict[str, Any]]: - """Filtering and grouping happen in the server: a busy service logs far more inserts - into other tables than any row limit could hold.""" - skip = f" AND query_id NOT IN ({_literals(excluded)})" if excluded else "" - rows = scope.rows( - "SELECT materialize(hostName()) AS host, user, " - "mapContains(Settings, 'async_insert') AS has_async, Settings['async_insert'] AS async_value, " - "mapContains(Settings, 'wait_for_async_insert') AS has_wait, " - "Settings['wait_for_async_insert'] AS wait_value, " - "sumIf(written_rows, type = 'QueryFinish') AS finished_rows, uniqExact(query_id) AS queries " - f"FROM {scope.log()} WHERE query_start_time >= now() - INTERVAL {_WINDOW_MINUTES} MINUTE " - "AND query_kind = 'Insert' AND is_initial_query = 1 " - f"AND type IN ('QueryStart', 'QueryFinish') AND {_inserts_target_sql(request)}{skip} " - "GROUP BY host, user, has_async, async_value, has_wait, wait_value " - f"LIMIT {_LOG_LIMIT}" - ) - if len(rows) >= _LOG_LIMIT: - raise RuntimeError("Writer groups exceed the bounded 60-minute inspection window") - writers = [] - for host, user, has_async, async_value, has_wait, wait_value, finished, queries in rows: - settings = {"async_insert": str(async_value)} if has_async else {} - if has_wait: - settings["wait_for_async_insert"] = str(wait_value) - writers.append( - {"host": str(host), "user": str(user), "settings": settings, "rows": int(finished)} - ) - writers[-1]["queries"] = int(queries) - return writers - - -def _inserts_target_sql(request: RebuildRequest) -> str: - """Direct INSERTs into the source table; QueryStart rows may list no tables, so the - statement prefix identifies direct writes and excludes materialized-view side effects.""" - db, table = request.database, request.source.name - target = f"{db}.{table}" - named = ( - "replaceRegexpAll(extract(query, " - r"'(?i)^\\s*INSERT\\s+INTO\\s+(?:TABLE\\s+)?((?:[`\"]?\\w+[`\"]?\\.)?[`\"]?\\w+[`\"]?)'" - "), '[`\"]', '')" - ) - return ( - f"if({named} != '', {named} = {_literal(target)} OR ({named} = {_literal(table)} " - f"AND current_database = {_literal(db)}), has(tables, {_literal(target)}) " - r"AND match(query, '(?i)^\\s*INSERT\\b'))" - ) - - -def _queued_writers(scope: _Scope, request: RebuildRequest) -> dict[tuple[str, str], dict]: - """Queued inserts the log cannot vouch for, with settings from their live process.""" - rows = scope.rows( - "SELECT materialize(hostName()), entries.query_id FROM " - + scope.table("asynchronous_inserts") - + f" WHERE database = {_literal(request.database)} AND table = {_literal(request.source.name)}" - ) - queued = {(str(host), str(qid)) for host, query_ids in rows for qid in query_ids} - logged = _logged_users(scope, queued) - unknown = {key for key in queued if not logged.get(key, (None, False))[1]} - active = _queued_settings(scope, unknown) - writers = {} - for key in sorted(unknown): - user = logged.get(key, ("", False))[0] - writer = {"host": key[0], "user": user, "settings": {}, "rows": 0, "queries": 1} - writer.update(active.get(key, {})) - if not writer["settings"]: - writer["queued_without_settings"] = True - writers[key] = writer - return writers - - -def _logged_users(scope: _Scope, queued: set[tuple[str, str]]) -> dict[tuple[str, str], tuple]: - """(user, logged with per-query settings) for each queued insert the log has seen.""" - if not queued: - return {} - rows = scope.rows( - f"SELECT materialize(hostName()) AS host, query_id, any(user), max(notEmpty(Settings)) " - f"FROM {scope.log()} WHERE query_id IN ({_literals({qid for _, qid in queued})}) " - "AND is_initial_query = 1 GROUP BY host, query_id" - ) - return { - (str(host), str(qid)): (str(user), bool(proven)) - for host, qid, user, proven in rows - if (str(host), str(qid)) in queued - } - - -def _literals(values: set[str]) -> str: - return ", ".join(_literal(value) for value in sorted(values)) - - -def _queued_settings(scope: _Scope, missing: set[tuple[str, str]]) -> dict: - if not missing: - return {} - ids = ", ".join(_literal(query_id) for _, query_id in sorted(missing)) - rows = scope.rows( - "SELECT materialize(hostName()), query_id, user, Settings FROM " - + scope.table("processes") - + " WHERE is_initial_query = 1 AND query_kind = 'Insert'" - + f" AND query_id IN ({ids})" - ) - return { - (str(host), str(query_id)): {"user": str(user), "settings": dict(settings)} - for host, query_id, user, settings in rows - if (str(host), str(query_id)) in missing - } - - -def _writer_findings( - scope: _Scope, request: RebuildRequest, writers: list[dict[str, Any]] -) -> list[PreflightFinding]: - findings = [ - PreflightFinding( - "insert_rate", - "warning", - "Recent completed INSERT throughput is reported; async fire-and-forget writes may have zero written_rows.", - { - "window_minutes": _WINDOW_MINUTES, - "writer_queries": sum(writer["queries"] for writer in writers), - }, - ) - ] - if not writers: - findings.append( - PreflightFinding( - "writer_history_limited", - "warning", - "No writers found in the bounded query-log window; this does not establish that none exist.", - ) - ) - return findings - profiles = _profile_data(scope) - for writer in writers: - effective = _effective_settings(writer, profiles) - if writer.get("queued_without_settings") and effective != ("1", "0"): - effective = None # Safe profile defaults cannot prove unlogged per-query overrides. - if effective is None: - findings.append( - PreflightFinding( - "writer_settings_unknown", - "warning" if request.allow_unacknowledged_async_loss else "refusal", - "Cannot prove this writer's async/wait settings; treat it as possible unacknowledged async loss.", - { - "host": writer["host"], - "user": writer["user"], - "acknowledged_loss": request.allow_unacknowledged_async_loss, - }, - ) - ) - elif effective == ("1", "0"): - severity = "warning" if request.allow_unacknowledged_async_loss else "refusal" - findings.append( - PreflightFinding( - "unacknowledged_async_writer", - severity, - "Writer uses async_insert=1 with wait_for_async_insert=0; queued unacknowledged writes can be lost at swap.", - { - "host": writer["host"], - "user": writer["user"], - "acknowledged_loss": request.allow_unacknowledged_async_loss, - }, - ) - ) - return findings - - -def _profile_data(scope: _Scope) -> dict[str, dict[str, Any]]: - tables = { - "profiles": "SELECT materialize(hostName()), name, apply_to_all, apply_to_list, apply_to_except FROM " - + scope.table("settings_profiles"), - "elements": "SELECT materialize(hostName()), profile_name, user_name, role_name, setting_name, value, inherit_profile FROM " - + scope.table("settings_profile_elements"), - "users": "SELECT materialize(hostName()), name, default_roles_all, default_roles_list, default_roles_except FROM " - + scope.table("users"), - "grants": "SELECT materialize(hostName()), user_name, role_name, granted_role_name FROM " - + scope.table("role_grants"), - "defaults": "SELECT materialize(hostName()), name, default FROM " - + scope.table("settings") - + " WHERE name IN ('async_insert', 'wait_for_async_insert')", - } - result: dict[str, dict[str, Any]] = {} - for kind, sql in tables.items(): - for host, *record in scope.rows(sql): - result.setdefault(str(host), {name: [] for name in tables})[kind].append(tuple(record)) - return result - - -def _effective_settings( - writer: dict[str, Any], by_host: dict[str, dict[str, Any]] -) -> tuple[str, str] | None: - host = by_host.get(writer["host"]) - if host is None: - return None - user = writer["user"] - identity = next((row for row in host["users"] if row[0] == user), None) - if identity is None: - return None - # Active roles are not recorded in query_log. Include every granted role and - # refuse conflicting unlogged values rather than assuming default roles. - roles = _possible_roles(user, host["grants"]) - applicable = _applicable_profiles(user, roles, host) - values = {key: set() for key in _SETTING_NAMES} - for profile in applicable: - _collect_profile(profile, (host["elements"], values), set()) - _collect_assigned(user, roles, (host["elements"], values)) - defaults = dict(host["defaults"]) - for name in _SETTING_NAMES: - if not values[name] and name in defaults: - values[name].add(str(defaults[name])) - explicit = writer["settings"] - result = [] - for name in _SETTING_NAMES: - if name in explicit: - result.append(str(explicit[name])) - elif len(values[name]) == 1: - result.append(next(iter(values[name]))) - else: - return None - return result[0], result[1] - - -def _possible_roles(user: str, grants: list[tuple[Any, ...]]) -> set[str]: - roles = {str(granted) for grantee, role, granted in grants if grantee == user} - pending = list(roles) - while pending: - role = pending.pop() - for grantee, parent, granted in grants: - if grantee is None and parent == role and granted not in roles: - roles.add(str(granted)) - pending.append(str(granted)) - return roles - - -def _applicable_profiles(user: str, roles: set[str], host: dict[str, Any]) -> set[str]: - principals = roles | {user} - matched = set() - for name, all_users, included, excluded in host["profiles"]: - if (all_users and not principals.issubset(set(excluded))) or principals.intersection( - included - ): - matched.add(str(name)) - return matched - - -def _collect_assigned(user: str, roles: set[str], state: tuple) -> None: - elements, values = state - for name, assigned_user, assigned_role, setting, value, parent in elements: - if assigned_user != user and assigned_role not in roles: - continue - if parent: - _collect_profile(str(parent), (elements, values), set()) - if setting in values and value is not None: - values[setting].add(str(value)) - - -def _collect_profile(profile: str, state: tuple, visited: set[str]) -> None: - elements, values = state - if profile in visited: - raise RuntimeError(f"Settings profile inheritance cycle at {profile!r}") - visited.add(profile) - for name, user, role, setting, value, parent in elements: - if name != profile or user is not None or role is not None: - continue - if parent: - _collect_profile(str(parent), (elements, values), visited) - if setting in values and value is not None: - values[setting].add(str(value)) - visited.remove(profile) - - -def _dedup_findings(scope: _Scope, request: RebuildRequest) -> list[PreflightFinding]: - if "Replicated" not in request.source.engine and "Shared" not in request.source.engine: - return [] - rows = scope.rows( - "SELECT value FROM system.settings WHERE name = 'deduplicate_blocks_in_dependent_materialized_views'" - ) - if not rows: - raise RuntimeError("Cannot inspect dependent materialized-view deduplication") - if str(rows[0][0]) == "0": - return [ - PreflightFinding( - "dependent_mv_dedup", - "warning", - "Dependent materialized-view block deduplication is disabled in this session; inspect writer profiles as well.", - ) - ] - return [] - - -def _engine_note(request: RebuildRequest) -> str: - engine = request.target.engine.split("(", 1)[0] - if engine in _REPLACING: - return "ReplacingMergeTree-family merges can collapse duplicate copies sharing the sorting key; visibility before merges is not deduplicated." - return "Row copy may keep a small number of duplicates. Collapsing and VersionedCollapsing engines do not guarantee elimination of identical positive rows." - - -def _literal(value: str) -> str: - return "'" + value.replace("\\", "\\\\").replace("'", "\\'") + "'" - - -def _normalized(value: str) -> str: - return "".join(_tokens(value)) diff --git a/ch_migrate/rebuild_types.py b/ch_migrate/rebuild_types.py deleted file mode 100644 index a585136..0000000 --- a/ch_migrate/rebuild_types.py +++ /dev/null @@ -1,63 +0,0 @@ -"""Shared online-rebuild values; credentials never enter persisted state.""" - -from __future__ import annotations - -from dataclasses import dataclass -from typing import Any, Callable - -from ch_migrate.introspect import TableDefinition -from ch_migrate.version_table import VersionTableState -from ch_migrate.waiting_types import WaitBudget - -# Pause between the copying view reaching every replica and the drain barrier on Cloud. -DEFAULT_GRACE_SECONDS = 10.0 - - -@dataclass(frozen=True) -class RebuildOptions: - database: str - table: str - revision: str - generation: int - cluster: str | None = None - select: str | None = None - allow_unacknowledged_async_loss: bool = False - writers_on_this_service_only: bool = False - grace_seconds: float = DEFAULT_GRACE_SECONDS - - -@dataclass(frozen=True) -class RebuildDefinition: - database: str - table: str - source: TableDefinition - target: TableDefinition - create_sql: str - projection: str - columns: tuple[str, ...] - cluster: str | None - revision: str - generation: int - allow_unacknowledged_async_loss: bool = False - writers_on_this_service_only: bool = False - grace_seconds: float = DEFAULT_GRACE_SECONDS - - -@dataclass(frozen=True) -class HelperIdentity: - """How a rebuild proves a helper is its own: a chosen UUID, or (Cloud) a comment token.""" - - uuid: str | None = None - token: str | None = None - - -@dataclass(frozen=True) -class RebuildRuntime: - client: Any - control_factory: Callable[[], Any] - definition: RebuildDefinition - deployment: VersionTableState - budget: WaitBudget - record: dict - checkpoint: Callable[[], None] - refresh: Callable[[], dict | None] diff --git a/ch_migrate/scaffold.py b/ch_migrate/scaffold.py index 9ecbfa6..958e10f 100644 --- a/ch_migrate/scaffold.py +++ b/ch_migrate/scaffold.py @@ -1,4 +1,4 @@ -"""SQL-first guarded rebuilds and the deprecated EXCHANGE scaffold.""" +"""EXCHANGE TABLES migration scaffold generation.""" from __future__ import annotations @@ -18,24 +18,16 @@ def fetch_current_ddl(env_config: dict[str, Any], table_name: str) -> str | None Returns: DDL string or None if connection fails or table doesn't exist. """ - from clickhouse_connect.driver.binding import quote_identifier - from ch_migrate.connection import get_client - client = None try: client = get_client(env_config) db = env_config["database"] - result = client.query( - f"SHOW CREATE TABLE {quote_identifier(db)}.{quote_identifier(table_name)}" - ) + result = client.query(f"SHOW CREATE TABLE {db}.{table_name}") if result.result_rows: return result.result_rows[0][0] except Exception: return None - finally: - if client is not None: - client.close() return None @@ -72,28 +64,6 @@ def find_dependent_dictionaries(env_config: dict[str, Any], table_name: str) -> return [] -def generate_rebuild_sql(table_name: str, current_ddl: str | None = None) -> str: - """Start a portable replacement CREATE, or an explicitly commented offline placeholder.""" - from ch_migrate.rebuild_ddl import _quote - - header = ( - f"-- Guarded online rebuild of {table_name}.\n" - "-- Edit ORDER BY, the engine or other desired schema details here; keep the table name.\n" - "-- The rebuild copies and swaps while writers keep running.\n" - "-- Duplicate copies are possible inside the recorded short rebuild window.\n" - "-- Reverse this change with another forward rebuild migration.\n" - "-- {db} is replaced with the selected environment's database.\n\n" - ) - if current_ddl is None: - return header + ( - "-- Live DDL was unavailable. Replace these comments with one complete CREATE TABLE.\n" - f"-- Start from SHOW CREATE TABLE {{db}}.{_quote(table_name)}.\n" - f"-- CREATE TABLE {{db}}.{_quote(table_name)} (your column definitions)\n" - "-- ENGINE = your_engine ORDER BY your_sorting_key;\n" - ) - return header + _portable_rebuild_ddl(current_ddl, table_name).rstrip().rstrip(";") + ";\n" - - def _make_shadow_ddl(ddl: str, table_name: str) -> str: """Transform a CREATE TABLE statement into a shadow table version. @@ -277,25 +247,3 @@ def rewrite_migration_file( dict_names=dict_names, ) migration_path.write_text(new_content) - - -def _portable_rebuild_ddl(ddl: str, table_name: str) -> str: - from ch_migrate.rebuild_ddl import _header, _ident, _quote, _replace, _tokens, _upper - - tokens = _tokens(ddl) - if len(tokens) < 4 or tuple(_upper(token) for token in tokens[:2]) != ("CREATE", "TABLE"): - raise ValueError("Live DDL is not a CREATE TABLE") - start = 2 - changes = [] - if tuple(_upper(token) for token in tokens[2:5]) == ("IF", "NOT", "EXISTS"): - changes.append((tokens[2].start, tokens[4].end, "")) - start = 5 - end = start + 2 if tokens[start + 1].text == "." else start - if _ident(tokens[end]) != table_name: - raise ValueError("Live DDL names a different table") - _, uuid_span, cluster_span = _header(tokens, end + 1) - changes.append((tokens[start].start, tokens[end].end, f"{{db}}.{_quote(table_name)}")) - for span in (uuid_span, cluster_span): - if span: - changes.append((*span, "")) - return _replace(ddl, changes) diff --git a/ch_migrate/secrets.py b/ch_migrate/secrets.py index e90577e..b54a725 100644 --- a/ch_migrate/secrets.py +++ b/ch_migrate/secrets.py @@ -28,8 +28,7 @@ def _get_ssm_client(region: Optional[str] = None): # type: ignore[no-untyped-de import boto3 # type: ignore[import-not-found,import-untyped] except ImportError: raise ImportError( - "boto3 is required for SSM support. " - "Install with: pip install ch-migrate-cli[ssm]" + "boto3 is required for SSM support. " "Install with: pip install ch-migrate-cli[ssm]" ) if region: return boto3.client("ssm", region_name=region) diff --git a/ch_migrate/skills/ch-migrate/SKILL.md b/ch_migrate/skills/ch-migrate/SKILL.md index 04d9ab6..fee70a4 100644 --- a/ch_migrate/skills/ch-migrate/SKILL.md +++ b/ch_migrate/skills/ch-migrate/SKILL.md @@ -15,7 +15,7 @@ transactional DDL. **Install:** `uv tool install ch-migrate-cli` or `pip install ch-migrate-cli`. The command is `ch-migrate`; the import package is `ch_migrate`. -From 0.9, Python 3.10+ is required; 0.5.x is the last line for 3.9. Existing +From 0.6, Python 3.10+ is required; 0.5.x is the last line for 3.9. Existing projects must run `ch-migrate upgrade-env` to install the version-2 shim. The old environment is backed up, and repeating the command preserves that backup. It also records current script heads as `lint.gate_baseline`, preserving YAML diff --git a/ch_migrate/sql.py b/ch_migrate/sql.py index 983d74e..1c2f84e 100644 --- a/ch_migrate/sql.py +++ b/ch_migrate/sql.py @@ -126,7 +126,7 @@ def clean_database_error(message: str) -> str: def _database_reason(exc: Exception) -> str: """Preserve operator instructions from wait failures; trim driver noise only.""" - from ch_migrate.waiting_types import WaitingError + from ch_migrate.waiting.errors import WaitingError current: BaseException | None = exc seen: set[int] = set() diff --git a/ch_migrate/statements.py b/ch_migrate/statements.py index 3981d26..16e8455 100644 --- a/ch_migrate/statements.py +++ b/ch_migrate/statements.py @@ -9,17 +9,10 @@ from typing import Literal from ch_migrate.rebase import RevisionGraph -from ch_migrate.rebuild_types import DEFAULT_GRACE_SECONDS from ch_migrate.sql import split_statements MigrationDirection = Literal["upgrade", "downgrade"] _FSTRING_POSITIONS = sys.version_info >= (3, 12) -_REBUILD_OPTIONS = ( - "select", - "allow_unacknowledged_async_loss", - "writers_on_this_service_only", - "grace_seconds", -) @dataclass(frozen=True) @@ -29,21 +22,17 @@ class MigrationStatement: line: int comments: tuple[str, ...] direction: MigrationDirection - rebuild: RebuildCall | None = None - - -@dataclass(frozen=True) -class RebuildCall: - table: str - create_sql_path: str - select: str | None = None - allow_unacknowledged_async_loss: bool = False - writers_on_this_service_only: bool = False - grace_seconds: float = DEFAULT_GRACE_SECONDS + # False when the SQL only exists at run time; `sql` then holds the call's source. + readable: bool = True def migration_statements(path: Path) -> list[MigrationStatement]: - """Collect literal SQL in upgrade/downgrade bodies, preserving its source.""" + """Collect the SQL in upgrade/downgrade bodies, preserving its source. + + Literal SQL is returned as statements. A call that sends SQL known only at run time + (a variable, an expression) comes back as one unreadable statement, so callers can + insist on a waiver instead of passing it silently. + """ content = path.read_text() root = path.parent.parent if root.name == "migrations": @@ -63,10 +52,8 @@ def migration_statements(path: Path) -> list[MigrationStatement]: for call in calls: if _call_name(call) in ("read_sql", "run_sql"): result.extend(_file_statements(call, source, direction)) - elif _call_name(call) == "op.execute": + elif _sends_sql(call): result.extend(_inline_statements(call, source, direction)) - elif _call_name(call).split(".")[-1] == "rebuild_table": - result.extend(_rebuild_statements(call, source, direction)) return result @@ -99,14 +86,19 @@ def _call_name(call: ast.Call) -> str: return "" +def _sends_sql(call: ast.Call) -> bool: + """`op.execute`, and `execute`/`exec_driver_sql` on a connection such as `op.get_bind()`.""" + return isinstance(call.func, ast.Attribute) and call.func.attr in ("execute", "exec_driver_sql") + + def _file_statements( call: ast.Call, source: _Source, direction: MigrationDirection ) -> list[MigrationStatement]: - if not call.args or not isinstance(call.args[0], ast.Constant): - return [] - path_value = call.args[0].value + path_value = ( + call.args[0].value if call.args and isinstance(call.args[0], ast.Constant) else None + ) if not isinstance(path_value, str): - return [] + return [_unreadable(call, source, direction)] sql_path = source.path.parent.parent / "sql" / path_value statements = split_statements(sql_path.read_text()) filename = sql_path.relative_to(source.root).as_posix() @@ -123,7 +115,13 @@ def _inline_statements( return [] text = _literal_sql(call.args[0], source.content) if text is None: - return [] + # `op.execute(read_sql("file.sql"))`: the file's statements are collected from read_sql. + if isinstance(call.args[0], ast.Call) and _call_name(call.args[0]) in ( + "read_sql", + "run_sql", + ): + return [] + return [_unreadable(call, source, direction)] comments = _python_comments(source.lines, call.lineno) filename = source.path.relative_to(source.root).as_posix() result = [] @@ -134,51 +132,18 @@ def _inline_statements( return result -def _rebuild_statements( +def _unreadable( call: ast.Call, source: _Source, direction: MigrationDirection -) -> list[MigrationStatement]: - arguments = dict(zip(("table", "create_sql_path"), call.args)) - arguments.update({item.arg: item.value for item in call.keywords}) - values = {} - for name in ("table", "create_sql_path", *_REBUILD_OPTIONS): - value = arguments.get(name) - if value is None: - continue - try: - values[name] = ast.literal_eval(value) - except (ValueError, TypeError): - if name == "table" and (literal := _literal_sql(value, source.content)) is not None: - values[name] = literal - else: - raise ValueError( - f"{source.path}:{call.lineno}: rebuild {name} must be statically readable" - ) - if not all(isinstance(values.get(name), str) for name in ("table", "create_sql_path")): - raise ValueError( - f"{source.path}:{call.lineno}: rebuild requires literal table and SQL path" - ) - for name in ("allow_unacknowledged_async_loss", "writers_on_this_service_only"): - if not isinstance(values.get(name, False), bool): - raise ValueError(f"rebuild {name} must be an explicit boolean") - rebuild = RebuildCall(**values) - sql_path = source.path.parent.parent / "sql" / rebuild.create_sql_path - statements = split_statements(sql_path.read_text()) - if len(statements) != 1: - raise ValueError(f"{sql_path}: rebuild requires exactly one replacement CREATE TABLE") - statement = statements[0] - return [ - MigrationStatement( - statement.sql, - sql_path.relative_to(source.root).as_posix(), - statement.line, - statement.comments, - direction, - rebuild, - ) - ] +) -> MigrationStatement: + code = ast.get_source_segment(source.content, call) or ast.unparse(call) + filename = source.path.relative_to(source.root).as_posix() + comments = _python_comments(source.lines, call.lineno) + return MigrationStatement(code, filename, call.lineno, comments, direction, readable=False) def _literal_sql(node: ast.AST, content: str) -> str | None: + if _is_text_call(node): + return _literal_sql(node.args[0], content) if isinstance(node, ast.Constant) and isinstance(node.value, str): return node.value if not isinstance(node, ast.JoinedStr): @@ -199,6 +164,16 @@ def _literal_sql(node: ast.AST, content: str) -> str | None: return "".join(pieces) +def _is_text_call(node: ast.AST) -> bool: + """SQLAlchemy's `text("...")` or `sa.text("...")` around the SQL.""" + if not isinstance(node, ast.Call) or len(node.args) != 1: + return False + func = node.func + return (isinstance(func, ast.Name) and func.id == "text") or ( + isinstance(func, ast.Attribute) and func.attr == "text" + ) + + def _python_comments(lines: list[str], lineno: int) -> tuple[str, ...]: comments = [] index = lineno - 2 diff --git a/ch_migrate/templates/bootstrap/init_users.sql b/ch_migrate/templates/bootstrap/init_users.sql index b1aa6fe..d62fa6e 100644 --- a/ch_migrate/templates/bootstrap/init_users.sql +++ b/ch_migrate/templates/bootstrap/init_users.sql @@ -25,29 +25,8 @@ GRANT SELECT ON system.grants TO {project}_migration_role; GRANT SELECT ON system.databases TO {project}_migration_role; GRANT SELECT ON system.tables TO {project}_migration_role; GRANT SELECT ON system.mutations TO {project}_migration_role; -GRANT SELECT ON system.columns TO {project}_migration_role; GRANT CURRENT GRANTS(SELECT ON system.replicas) TO {project}_migration_role; -GRANT CURRENT GRANTS(SELECT ON system.clusters) TO {project}_migration_role; -GRANT CURRENT GRANTS(SELECT ON system.distributed_ddl_queue) TO {project}_migration_role; GRANT CURRENT GRANTS(SELECT ON system.parts) TO {project}_migration_role; -GRANT CURRENT GRANTS(SELECT ON system.parts_columns) TO {project}_migration_role; -GRANT CURRENT GRANTS(SELECT ON system.disks) TO {project}_migration_role; -GRANT CURRENT GRANTS(SELECT ON system.merge_tree_settings) TO {project}_migration_role; -GRANT CURRENT GRANTS(SELECT ON system.settings) TO {project}_migration_role; -GRANT CURRENT GRANTS(SELECT ON system.settings_profiles) TO {project}_migration_role; -GRANT CURRENT GRANTS(SELECT ON system.settings_profile_elements) TO {project}_migration_role; -GRANT CURRENT GRANTS(SELECT ON system.users) TO {project}_migration_role; -GRANT CURRENT GRANTS(SELECT ON system.role_grants) TO {project}_migration_role; -GRANT CURRENT GRANTS(SELECT ON system.query_log) TO {project}_migration_role; -GRANT CURRENT GRANTS(SELECT ON system.asynchronous_inserts) TO {project}_migration_role; -GRANT CURRENT GRANTS(SELECT ON system.asynchronous_insert_log) TO {project}_migration_role; -GRANT CURRENT GRANTS(SELECT ON system.part_log) TO {project}_migration_role; -GRANT CURRENT GRANTS(SELECT ON system.processes) TO {project}_migration_role; -GRANT CURRENT GRANTS(KILL QUERY ON *.*) TO {project}_migration_role; -GRANT CURRENT GRANTS(SYSTEM FLUSH LOGS ON *.*) TO {project}_migration_role; -GRANT CURRENT GRANTS(SYSTEM RELOAD DICTIONARY ON *.*) TO {project}_migration_role; -GRANT CURRENT GRANTS(SYSTEM FLUSH ASYNC INSERT QUEUE ON *.*) TO {project}_migration_role; -GRANT CURRENT GRANTS(SYSTEM MERGES ON {db}.*) TO {project}_migration_role; GRANT CURRENT GRANTS(SYSTEM SYNC REPLICA ON {db}.*) TO {project}_migration_role; GRANT CURRENT GRANTS(SYSTEM SYNC DATABASE REPLICA ON {db}.*) TO {project}_migration_role; GRANT CURRENT GRANTS(CLUSTER ON *.*) TO {project}_migration_role; diff --git a/ch_migrate/ui.py b/ch_migrate/ui.py index 08c0ff9..803d98d 100644 --- a/ch_migrate/ui.py +++ b/ch_migrate/ui.py @@ -32,8 +32,7 @@ class Failure(click.ClickException): """A command error that click shows as `✗ message` and then the hints. - `format_message()` keeps the message and hints joined as one plain string, - which is what `--json` callers get as the error. + `format_message()` keeps the message and hints joined as one plain string. """ def __init__(self, message: str, *hints: str) -> None: @@ -47,14 +46,14 @@ def show(self, file: IO[Any] | None = None) -> None: hint(line, stderr=True) -def step(message: str) -> None: +def step(message: str, *, stderr: bool = False) -> None: """Something is starting or in progress.""" - out.print(_marked("→ ", "cyan", message)) + (err if stderr else out).print(_marked("→ ", "cyan", message)) -def success(message: str) -> None: +def success(message: str, *, stderr: bool = False) -> None: """Something finished as intended.""" - out.print(_marked("✓ ", "green", message)) + (err if stderr else out).print(_marked("✓ ", "green", message)) def detail(message: str, *, stderr: bool = False) -> None: diff --git a/ch_migrate/version_table.py b/ch_migrate/version_table.py index 1476a6c..5a9b2dd 100644 --- a/ch_migrate/version_table.py +++ b/ch_migrate/version_table.py @@ -10,8 +10,6 @@ from clickhouse_connect.cc_sqlalchemy.sql.ddlcompiler import ChDDLCompiler from clickhouse_connect.driver.binding import quote_identifier -CLOUD_CLUSTER = "default" - @dataclass(frozen=True) class VersionTableState: @@ -32,15 +30,17 @@ def health_cluster(self) -> str | None: return None return self.cluster or (self.database if self.database_engine == "Replicated" else None) - @property - def replica_cluster(self) -> str | None: - """Cluster whose replicas a rebuild must inspect; Cloud names its replicas `default`.""" - if self.database_engine.startswith("Shared"): - return CLOUD_CLUSTER - return self.health_cluster - def new_engine(self): - return migration_state_engine(self, "alembic_version", "version_num") + if self.database_engine.startswith("Shared"): + return MergeTree(order_by="version_num") + if self.database_engine == "Replicated": + return ReplicatedMergeTree(order_by="version_num") + if self.cluster: + path = ( + f"/clickhouse/ch_migrate/{quote(self.database, safe='')}/{{shard}}/alembic_version" + ) + return ReplicatedMergeTree(order_by="version_num", zk_path=path, replica="{replica}") + return MergeTree(order_by="version_num") def warning(self) -> str | None: replicated = self.database_engine == "Replicated" or bool(self.cluster) @@ -73,18 +73,6 @@ class VersionTableMutationError(RuntimeError): """An unfinished version-table mutation has a server-reported failure.""" -def migration_state_engine(state: VersionTableState, table: str, order_by: str): - """Apply one deployment policy to version state and durable migration journals.""" - if state.database_engine.startswith("Shared"): - return MergeTree(order_by=order_by) - if state.database_engine == "Replicated": - return ReplicatedMergeTree(order_by=order_by) - if state.cluster: - path = f"/clickhouse/ch_migrate/{quote(state.database, safe='')}/{{shard}}/{table}" - return ReplicatedMergeTree(order_by=order_by, zk_path=path, replica="{replica}") - return MergeTree(order_by=order_by) - - def inspect_version_table(client, database: str, cluster: str | None = None) -> VersionTableState: rows = client.query( "SELECT engine FROM system.databases WHERE name = {db:String}", @@ -129,12 +117,8 @@ def assert_version_mutations_healthy(client, state: VersionTableState) -> None: settings=settings, ).result_rows if failed: - from ch_migrate.waiting_sql import sql_string - host, mutation, reason = failed[0] raise VersionTableMutationError( f"Version-table mutation {mutation} on {host} failed: {reason}. " - "ch-migrate did not kill it.\n" - f"Operator only on {host}: KILL MUTATION WHERE database = {sql_string(state.database)} " - f"AND table = 'alembic_version' AND mutation_id = {sql_string(mutation)}" + "Resolve the failed mutation manually before retrying; ch-migrate did not kill it." ) diff --git a/ch_migrate/waiting.py b/ch_migrate/waiting.py deleted file mode 100644 index 9fe40dd..0000000 --- a/ch_migrate/waiting.py +++ /dev/null @@ -1,506 +0,0 @@ -"""Journal actual cursor writes and wait before Alembic completes a revision.""" - -from __future__ import annotations - -import hashlib -import json -import sys -from contextlib import contextmanager -from functools import wraps -from pathlib import Path -from uuid import uuid4 - -from alembic.script import ScriptDirectory -from clickhouse_connect.driver.exceptions import DatabaseError -from sqlalchemy import event -from sqlalchemy.schema import CreateTable - -from ch_migrate.classify import classify -from ch_migrate.hooks import run_hooks -from ch_migrate.idempotency import classify_idempotency -from ch_migrate.introspect import ColumnDefinition, Schema, TableDefinition, _parse_order_by -from ch_migrate.statements import migration_statements -from ch_migrate.waiting_ddl import DistributedDDLWaiter -from ch_migrate.waiting_mutations import MutationWaiter -from ch_migrate.waiting_sql import ( - bind_statement, - is_session_or_read, - modifies_ttl, - mutation_sql, - qualified_table, - query_setting, - query_settings, - remove_query_settings, - sql_string, - statement_cluster, - statement_digest, - statement_table, -) -from ch_migrate.waiting_store import StepKey, WaitingJournal -from ch_migrate.waiting_types import UnknownOutcome, WaitBudget, WaitingError -from ch_migrate.waiting_versions import VersionWrites - - -class MigrationWaiter: - """One connection, one invocation deadline, and ordered durable receipts.""" - - def __init__(self, connection, state, timeout=None): - self.connection, self.state = connection, state - self.client = connection.connection.dbapi_connection.client - self.budget = WaitBudget(timeout) - self.journal = WaitingJournal(connection, state, self.budget) - self.mutations = MutationWaiter(self.client, state, self.budget) - self.ddl = DistributedDDLWaiter(self.client, self.budget) - self.versions = VersionWrites(self) - self.progress_stream = sys.stderr - self.revision = None - self.generation = self.position = self._internal = 0 - self.fingerprint = "" - self._pending = {} - self._batch_key = None - self._listeners = [] - self._scripts = None - - def install(self, config) -> None: - self._scripts = ScriptDirectory.from_config(config) - for target, name, listener, options in ( - (self.connection, "before_cursor_execute", self._before, {"retval": True}), - (self.connection, "after_cursor_execute", self._after, {}), - (self.connection.engine, "handle_error", self._error, {}), - ): - event.listen(target, name, listener, **options) - self._listeners.append((target, name, listener)) - - def close(self) -> None: - self.budget.finish() - for target, name, listener in reversed(self._listeners): - event.remove(target, name, listener) - self._listeners.clear() - self.revision = None - - @contextmanager - def internal(self): - self._internal += 1 - try: - yield - finally: - self._internal -= 1 - - def reserve(self, payload: dict) -> tuple[StepKey, dict, bool]: - self.position += 1 - key = StepKey(self.revision, self.generation, self.position) - previous = self.journal.read(key) - payload["fingerprint"] = self.fingerprint - if previous is None: - payload["token"] = "chm_mutation_" + uuid4().hex - return key, payload, True - retryable = ( - previous.get("phase") == "rejected" - and previous.get("repeat_safe") - and payload.get("repeat_safe") - ) - if previous["digest"] != payload["digest"] and not retryable: - raise UnknownOutcome( - f"Revision {key.revision} statement {key.position} changed after execution began; " - "statement not reissued. Reconcile its journal and the original SQL." - ) - if retryable: - payload["token"] = "chm_mutation_" + uuid4().hex - return key, payload if retryable else previous, bool(retryable) - - def validate_revision(self, revision: str, expected: str) -> None: - script = self._scripts.get_revision(revision) - if script is None or _fingerprint(Path(script.path)) != expected: - raise UnknownOutcome( - f"Revision {revision} changed during incomplete version bookkeeping; reconcile before resuming" - ) - - def unknown_outcome(self, key: StepKey, record: dict, error: Exception) -> UnknownOutcome: - receipt = record.get("receipt") or record.get("before_target") or {} - table = record.get("table") - if record.get("kind") == "version": - table = (self.state.database, "alembic_version") - receipt = next((step["receipt"] for step in record["steps"] if "receipt" in step), {}) - message = ( - f"{error}\nRevision {key.revision}, generation {key.generation}, statement {key.position}; " - f"token {record.get('token', receipt.get('token', 'unrecorded'))}; " - f"recorded target UUIDs {json.dumps(receipt.get('uuids', {}), sort_keys=True)}.\n" - "Stop migration runners. Establish the actual outcome independently; reconcile only this " - "journal key before rerunning. Do not stamp the revision or blindly delete its journal.\n" - f"Read-only inspection: SELECT sequence, payload FROM {self.journal.table} " - f"WHERE revision = {sql_string(key.revision)} AND generation = {key.generation} " - f"AND position = {key.position} ORDER BY sequence DESC;\n" - ) - if table: - source = ( - f"clusterAllReplicas({sql_string(receipt['cluster'])}, system.mutations)" - if receipt.get("cluster") - else "system.mutations" - ) - message += ( - "Read-only inspection: SELECT mutation_id, command, is_done, parts_to_do, latest_fail_reason " - f"FROM {source} WHERE database = {sql_string(table[0])} AND table = {sql_string(table[1])};" - ) - for name in ("ddl", "work_ddl"): - if record.get(name): - message += ( - "\nRead-only inspection: SELECT entry, host, port, status, exception_code, exception_text " - "FROM system.distributed_ddl_queue WHERE settings['log_comment'] = " - f"{sql_string(record[name]['token'])};" - ) - return UnknownOutcome(message) - - def steps(self, original, hooks): - def iterate(heads, context): - for step in original(heads, context): - if not hasattr(step, "revision"): - yield step # Native Alembic stamp steps have no migration body to track. - continue - revision = step.revision.revision - fingerprint = _fingerprint(Path(step.revision.path)) - if step.is_upgrade: - self._begin(revision, fingerprint) - else: - self.journal.invalidate(revision, fingerprint) - execute = step.migration_fn - step.migration_fn = self._revision_body(execute, hooks, revision) - try: - yield step - finally: - self.revision = None - - return iterate - - def ensure_version_table(self, table) -> None: - if not self.state.on_cluster: - return - scope = "__version_table__" - if self.state.table_engine and not self.journal.unfinished_scope(scope): - return - ddl = str(CreateTable(table, if_not_exists=True).compile(dialect=self.connection.dialect)) - self._begin(scope, statement_digest(ddl, None)) - try: - self.connection.exec_driver_sql(ddl) - finally: - self.revision = None - - def run_pre_hooks(self, hooks) -> None: - fingerprint = hashlib.sha256(json.dumps(hooks.pre_migrate).encode()).hexdigest() - self._begin("__batch__", fingerprint) - key = StepKey(self.revision, self.generation, 0) - if self.journal.read(key).get("finished"): - self.journal.invalidate(self.revision, fingerprint) - self._begin("__batch__", fingerprint) - key = StepKey(self.revision, self.generation, 0) - self._batch_key = key - run_hooks( - self.connection, - hooks.pre_migrate, - db=self.state.database, - phase="pre_migrate", - revision="all", - ) - boundary, record, _ = self.reserve( - {"kind": "boundary", "phase": "done", "digest": "pre-hooks-complete"} - ) - self.journal.write(boundary, record) - self.revision = None - - def finish_run(self) -> None: - if self._batch_key: - control = self.journal.read(self._batch_key) - control["finished"] = True - self.journal.write(self._batch_key, control) - - def _begin(self, revision: str, fingerprint: str) -> None: - self.revision, self.fingerprint, self.position = revision, fingerprint, 0 - self.generation = self.journal.begin(revision, fingerprint) - self._completed_targets = {} - for key, record in self.journal.records(revision, self.generation): - if ( - record["phase"] not in ("done", "rejected") - and record.get("fingerprint") != fingerprint - ): - raise UnknownOutcome( - f"Revision {revision} changed with unresolved work; statement not reissued" - ) - if ( - record["phase"] == "accepted" - and record["kind"] == "write" - and not record.get("ddl") - ): - self._complete(key, record) - if record["phase"] == "done" and record.get("table"): - self._completed_targets[tuple(record["table"])] = record.get("after_target") - - def _revision_body(self, execute, hooks, revision): - @wraps(execute) - def run(**kwargs): - execute(**kwargs) - run_hooks( - self.connection, - hooks.post_migrate, - db=self.state.database, - phase="post_migrate", - revision=revision, - ) - - return run - - def _before(self, connection, cursor, statement, parameters, context, executemany): - # SQLAlchemy's event signature is an external contract. - if self._internal or self.revision is None: - return statement, parameters - sample = parameters[0] if executemany and parameters else parameters - sql = bind_statement(statement, sample, self.client) - if is_session_or_read(sql): - return statement, parameters - if executemany: - return self._execute_many(statement, parameters, context.execution_options) - table = statement_table(sql, self.state.database) - payload = { - "kind": "write", - "phase": "intent", - "digest": statement_digest(statement, parameters), - "table": list(table) if table else None, - "repeat_safe": classify_idempotency(sql).status == "ok", - } - key, record, fresh = self.reserve(payload) - if not fresh and self._resume(key, record): - return "SELECT 1 WHERE 0", {} - cluster = statement_cluster(sql) - if cluster: - record["ddl"] = self.ddl.prepare(cluster, record["token"] + "_ddl") - record["caller_log_comment"] = query_setting(sql, "log_comment") - target = self._target(table, record, allow_missing=True) if table else None - if cluster and table and target is None and classify(sql).kind == "mutation": - raise WaitingError( - f"No reachable mutation target {table} in cluster {cluster}; nothing was submitted" - ) - record["before_target"] = target - kind = self._work_kind(sql, target, sample) if target else None - if kind: - token = record["token"] - record["kind"] = kind - record["receipt"] = self.mutations.prepare(target, token) - record["receipt"]["expected_markers"] = 1 - if kind == "ttl": - record.update(kind="ttl", work_submitted=False, cluster=statement_cluster(sql)) - statement = query_settings( - statement, {"materialize_ttl_after_modify": "0", "alter_sync": "0"} - ) - else: - statement = mutation_sql(statement, token) - if record.get("ddl"): - statement, settings = self._ddl_submission(statement, record["ddl"]) - previous = self._statement_settings(context, settings) - if record["caller_log_comment"] is None: - record["caller_log_comment"] = previous.get("log_comment") - self.journal.write(key, record) # Acknowledged intent MUST precede submission. - self._pending[id(context)] = (key, record) - return statement, parameters - - def _execute_many(self, statement, parameters, execution_options): - # Re-enter the single-statement cursor path, preserving statement order - # and an independently durable intent for each parameter set. - for binding in parameters: - self.connection.exec_driver_sql( - statement, binding, execution_options=execution_options - ).close() - return "SELECT 1 WHERE 0", [{}] - - def _after(self, connection, cursor, statement, parameters, context, executemany): - pending = self._pending.pop(id(context), None) - if pending is None: - return - key, record = pending - record["phase"] = "accepted" - self.journal.write(key, record) - if record.get("ddl") or record["kind"] in ("mutation", "ttl"): - self._wait_work(key, record) - self._complete(key, record) - - def _error(self, context): - if context.connection is not self.connection: - return - pending = self._pending.pop(id(context.execution_context), None) - if pending is None: - return - key, record = pending - # A server rejection of repeat-safe metadata can be repaired and retried. - # Transport loss and mutation submissions remain unknown, never replayed. - if ( - record["kind"] == "write" - and record["repeat_safe"] - and not record.get("ddl") - and isinstance(context.original_exception, DatabaseError) - and context.original_exception.code is not None - ): - record["phase"] = "rejected" - self.journal.write(key, record) - - def _resume(self, key: StepKey, record: dict) -> bool: - if record["phase"] == "done": - self._check_completed_target(record) - return True - if ( - record.get("ddl") - or record["kind"] == "mutation" - or (record["kind"] == "ttl" and record["phase"] == "accepted") - ): - print( - f"Resuming {key.revision} statement {key.position}; not reissuing SQL", - file=self.progress_stream, - ) - self._wait_work(key, record) - self._complete(key, record) - return True - if record["phase"] == "accepted": - self._complete(key, record) - return True - error = UnknownOutcome("Outcome unknown; statement not reissued; migration incomplete") - raise self.unknown_outcome(key, record, error) from error - - def _wait_work(self, key: StepKey, record: dict) -> None: - def checkpoint() -> None: - self.journal.write(key, record) - - try: - if record.get("ddl"): - self.ddl.wait(record["ddl"], checkpoint) - record["phase"] = "accepted" - checkpoint() - if record["kind"] == "ttl": - self._materialize_ttl(record, checkpoint) - if record["kind"] in ("mutation", "ttl"): - self._refresh_distributed_target(record) - self.mutations.wait(record["receipt"], checkpoint) - except UnknownOutcome as error: - raise self.unknown_outcome(key, record, error) from error - - def _materialize_ttl(self, record: dict, checkpoint) -> None: - if not record["work_submitted"]: - cluster = ( - f" ON CLUSTER {sql_string(record['cluster'])}" if record.get("cluster") else "" - ) - sql = f"ALTER TABLE {qualified_table(tuple(record['table']))}{cluster} MATERIALIZE TTL" - sql = mutation_sql(sql, record["receipt"]["token"]) - settings = {} - if record.get("ddl"): - record["work_ddl"] = self.ddl.prepare(record["cluster"], record["token"] + "_ttl") - sql, settings = self._ddl_submission(sql, record["work_ddl"]) - record["work_submitted"] = True - checkpoint() - with self.internal(): - self.connection.exec_driver_sql(sql, execution_options={"settings": settings}) - if record.get("work_ddl"): - self.ddl.wait(record["work_ddl"], checkpoint) - - def _refresh_distributed_target(self, record: dict) -> None: - if not record.get("ddl"): - return - actual = self._target(tuple(record["table"]), record) - if actual is None: - raise UnknownOutcome("Distributed mutation target disappeared; statement not reissued") - receipt = record["receipt"] - for host, uuid in receipt["uuids"].items(): - if actual["uuids"].get(host) != uuid: - raise UnknownOutcome( - f"Distributed mutation target was replaced on {host}; statement not reissued" - ) - receipt["uuids"] = actual["uuids"] - - def _target(self, table, record, allow_missing=False): - if record.get("ddl"): - return self.ddl.tables(table, record["ddl"]["cluster"], allow_missing) - return self.mutations.target(table) - - @staticmethod - def _ddl_submission(statement: str, receipt: dict) -> tuple[str, dict]: - settings = { - "distributed_ddl_task_timeout": 0, - "distributed_ddl_output_mode": "none", - "log_comment": receipt["token"], - } - return remove_query_settings(statement, set(settings)), settings - - @staticmethod - def _statement_settings(context, overrides: dict) -> dict: - previous = dict(context.execution_options.get("settings") or {}) - statement = getattr(context, "invoked_statement", None) - if statement is not None: - previous.update(statement.get_execution_options().get("settings") or {}) - context.invoked_statement = statement.execution_options( - settings={**previous, **overrides} - ) - context.execution_options = context.execution_options.union( - {"settings": {**previous, **overrides}} - ) - return previous - - def _complete(self, key: StepKey, record: dict) -> None: - if record.get("table"): - record["after_target"] = self._target(tuple(record["table"]), record) - self._completed_targets[tuple(record["table"])] = record["after_target"] - record["phase"] = "done" - self.journal.write(key, record) - - def _check_completed_target(self, record: dict) -> None: - if not record.get("table"): - return - expected = self._completed_targets.get(tuple(record["table"]), record.get("after_target")) - actual = self._target(tuple(record["table"]), record) - if ( - (expected is None) != (actual is None) - or expected is not None - and expected["uuids"] != actual["uuids"] - ): - raise UnknownOutcome( - "Completed statement's target was dropped or replaced; reconcile before skipping its SQL" - ) - - def _work_kind(self, sql: str, target: dict, parameters) -> str | None: - if not target["engine"].endswith("MergeTree"): - return None - rows = self.client.query( - "SELECT name, type FROM system.columns WHERE database = {db:String} AND table = {table:String}", - parameters={"db": target["database"], "table": target["table"]}, - ).result_rows - table = TableDefinition( - target["table"], - target["engine"], - columns=[ColumnDefinition(*row) for row in rows], - order_by=_parse_order_by(target["sorting_key"]) if target["sorting_key"] else [], - ) - schema = ( - None - if target.get("ddl_scope") - else Schema(database=target["database"], tables={target["table"]: table}) - ) - ttl = modifies_ttl(sql) - materialize = self._materializes_ttl(sql, parameters) if ttl else False - analyzed = ( - query_settings(sql, {"materialize_ttl_after_modify": "0"}) - if ttl and not materialize - else sql - ) - if classify(analyzed, schema).kind != "mutation": - return None - return "ttl" if ttl and materialize else "mutation" - - def _materializes_ttl(self, sql: str, parameters) -> bool: - value = query_setting(sql, "materialize_ttl_after_modify") - if value is None: - value = str(self.client.command("SELECT getSetting('materialize_ttl_after_modify')")) - if value.startswith("{") and isinstance(parameters, dict): - value = str(parameters[value[1:].split(":", 1)[0]]) - if value.lower() not in ("0", "1", "true", "false"): - raise WaitingError("materialize_ttl_after_modify must resolve to a boolean value") - return value.lower() in ("1", "true") - - -def _fingerprint(path: Path) -> str: - digest = hashlib.sha256(path.read_bytes()) - for statement in migration_statements(path): - digest.update(statement.sql.encode()) - return digest.hexdigest() diff --git a/ch_migrate/waiting/__init__.py b/ch_migrate/waiting/__init__.py new file mode 100644 index 0000000..cbf3221 --- /dev/null +++ b/ch_migrate/waiting/__init__.py @@ -0,0 +1,13 @@ +"""Waiting for the work migrations start: mutations and ON CLUSTER DDL. + +`up` doesn't record a migration until the mutations and distributed DDL its statements +started have finished. `MigrationWaiter` (waiter.py) hooks the migration connection's +cursor events: before each write it bounds ClickHouse's own distributed DDL wait, and +after it, mutations.py waits until the database has no unfinished mutations on any +replica that must apply them. display.py shows the progress; budget.py holds the +invocation's `--timeout`. +""" + +from ch_migrate.waiting.waiter import MigrationWaiter + +__all__ = ["MigrationWaiter"] diff --git a/ch_migrate/waiting/budget.py b/ch_migrate/waiting/budget.py new file mode 100644 index 0000000..6da2baa --- /dev/null +++ b/ch_migrate/waiting/budget.py @@ -0,0 +1,47 @@ +"""One waiting deadline per invocation, and the poll between checks.""" + +from __future__ import annotations + +import time +from dataclasses import dataclass, field + +from ch_migrate.waiting.display import Progress, WaitDisplay +from ch_migrate.waiting.errors import WaitTimeout + + +@dataclass +class WaitBudget: + timeout: float | None = None + started: float = field(default_factory=time.monotonic) + poll_interval: float = 0.2 + display: WaitDisplay = field(default_factory=WaitDisplay) + + @property + def elapsed(self) -> float: + return time.monotonic() - self.started + + def check(self, pending: str) -> None: + if self.timeout is not None and self.elapsed >= self.timeout: + self.finish() + raise WaitTimeout( + f"Timed out after {self.elapsed:.1f}s; server work continues; " + f"migration incomplete. Pending: {pending}", + pending, + self.elapsed, + ) + + def pause(self, pending: str, progress: Progress | None = None) -> None: + """Show `progress` (or the pending text), then sleep one poll within the deadline.""" + self.check(pending) + self.display.update(progress if progress is not None else pending) + remaining = self.poll_interval + if self.timeout is not None: + remaining = min(remaining, max(0, self.timeout - self.elapsed)) + time.sleep(remaining) + self.check(pending) + + def complete(self, message: str) -> None: + self.display.complete(message) + + def finish(self) -> None: + self.display.stop() diff --git a/ch_migrate/waiting/display.py b/ch_migrate/waiting/display.py new file mode 100644 index 0000000..7dcbbf1 --- /dev/null +++ b/ch_migrate/waiting/display.py @@ -0,0 +1,265 @@ +"""What `up` shows while it waits for mutations. + +Waiters hand over what the server reports; this module only draws it, and nothing +here decides whether work is done. On a terminal each wait is a live bar on +stderr, weighted by bytes when part sizes are known and labelled with parts. Piped +output, for CI logs and agents, gets plain lines instead: one when a wait starts, +one at each 10% step or every 30 seconds, and one when it finishes. +""" + +from __future__ import annotations + +import time +from dataclasses import dataclass, field +from typing import Union + +from rich.console import Console, Group +from rich.live import Live +from rich.progress_bar import ProgressBar +from rich.table import Table +from rich.text import Text + +from ch_migrate import ui + +PIPED_INTERVAL = 30.0 +PIPED_STEPS = 10 +BAR_WIDTH = 38 + +# What a waiter reports: a mutation, or a plain status line. +Progress = Union["MutationProgress", str] + + +class WaitDisplay: + """Draws one wait at a time; `complete` prints its result line, `stop` clears it.""" + + def __init__(self, console: Console | None = None) -> None: + self.console = console or ui.err + self.migration = "" + self._wait: _Wait | None = None + self._live: Live | None = None + self._parts_only = False + + def start_migration(self, label: str) -> None: + self.migration = label + + @property + def wants_bytes(self) -> bool: + """Only the terminal bar shows bytes, so piped runs skip the size lookup.""" + return self._terminal and not self._parts_only + + def update(self, progress: Progress) -> None: + wait = self._wait_for(progress) + wait.observe(progress) + if self._terminal: + self._draw_live(wait) + else: + self._print_piped(wait) + + def parts_only(self, reason: str) -> None: + """Part sizes can't be read; say so once, then count parts.""" + if self._parts_only: + return + self._parts_only = True + self.console.print( + Text.assemble(("! ", "yellow"), (f"Progress counts parts only: {reason}", "yellow")) + ) + + def complete(self, fallback: str) -> None: + wait = self._wait + self.stop() + if wait is not None: + self.console.print( + Text.assemble(("✓ ", "green"), wait.summary(self.migration, fallback)) + ) + + def stop(self) -> None: + if self._live is not None: + self._live.stop() + self._live = None + self._wait = None + + def _wait_for(self, progress: Progress) -> _Wait: + key = _key(progress) + if self._wait is None or self._wait.key != key: + self.stop() + self._wait = _Wait(key, started=time.monotonic()) + return self._wait + + @property + def _terminal(self) -> bool: + # A dumb terminal can't redraw in place, so it gets the piped lines. + return self.console.is_terminal and not self.console.is_dumb_terminal + + def _draw_live(self, wait: _Wait) -> None: + if self._live is None: + self._live = Live(console=self.console, auto_refresh=False, transient=True) + self._live.start() + self._live.update(wait.renderable(), refresh=True) + + def _print_piped(self, wait: _Wait) -> None: + line = wait.piped_line(time.monotonic()) + if line is not None: + self.console.print(Text(" " + line)) + + +@dataclass(frozen=True) +class MutationProgress: + """The mutation a statement started, on every host that hasn't finished it.""" + + table: str + mutation: str + hosts: tuple[HostParts, ...] + + +@dataclass(frozen=True) +class HostParts: + """One host's remaining parts. `sizes` maps each remaining part to its bytes, if known.""" + + host: str + names: frozenset[str] + count: int + sizes: dict[str, int] | None = None + + @property + def bytes_left(self) -> int | None: + if self.sizes is None or not self.names.issubset(self.sizes): + return None + return sum(self.sizes[name] for name in self.names) + + +@dataclass +class _Wait: + key: str + started: float + latest: Progress = "" + parts_total: dict[str, int] = field(default_factory=dict) + bytes_total: dict[str, int] = field(default_factory=dict) + printed_at: float | None = None + printed_step: int = -1 + + def observe(self, progress: Progress) -> None: + self.latest = progress + if isinstance(progress, MutationProgress): + for host in progress.hosts: + self.parts_total[host.host] = max(self.parts_total.get(host.host, 0), host.count) + if host.host not in self.bytes_total and host.bytes_left is not None: + self.bytes_total[host.host] = host.bytes_left + + def renderable(self): + elapsed = _duration(time.monotonic() - self.started) + progress = self.latest + if isinstance(progress, str): + return Text(f" waiting: {progress} · {elapsed}", style="dim") + rows = [self._host_row(host, len(progress.hosts) > 1) for host in progress.hosts] + return Group(*rows) + + def piped_line(self, now: float) -> str | None: + step = int(self._fraction() * PIPED_STEPS) + due = self.printed_at is None or now - self.printed_at >= PIPED_INTERVAL + if not due and step <= self.printed_step: + return None + first = self.printed_at is None + self.printed_at, self.printed_step = now, step + elapsed = "" if first else f" · {_duration(now - self.started)}" + return self._piped_text() + elapsed + + def summary(self, migration: str, fallback: str) -> str: + elapsed = _duration(time.monotonic() - self.started) + progress = self.latest + if isinstance(progress, str): + return f"{migration or fallback} waited for {progress} · {elapsed}" + return f"{migration or progress.table} {self._mutation_totals(progress)} · {elapsed}" + + def _host_row(self, host: HostParts, named: bool): + total = self.parts_total.get(host.host, host.count) + parts = f"{total - host.count}/{total} parts" + left, size_total = host.bytes_left, self.bytes_total.get(host.host) + if left is None or not size_total: + return _bar_row(host.host if named else "", _bar(total - host.count, total), parts) + done = size_total - left + stats = ( + f"{_sizes(done, size_total)} · {parts} · {_duration(time.monotonic() - self.started)}" + ) + if eta := self._eta(done, left): + stats += f" · ~{eta} left" + return _bar_row(host.host if named else "", _bar(done, size_total), stats) + + def _eta(self, done: int, left: int) -> str: + elapsed = time.monotonic() - self.started + if done <= 0 or elapsed < 2: + return "" + return _duration(left / (done / elapsed)) + + def _fraction(self) -> float: + progress = self.latest + if isinstance(progress, str): + return 0.0 + total = sum(self.parts_total.values()) + left = sum(host.count for host in progress.hosts) + return (total - left) / total if total else 0.0 + + def _piped_text(self) -> str: + progress = self.latest + if isinstance(progress, str): + return f"waiting: {progress}" + total = sum(self.parts_total.values()) + done = total - sum(host.count for host in progress.hosts) + replicas = f" on {len(progress.hosts)} replicas" if len(progress.hosts) > 1 else "" + return f"waiting on {_named(progress.mutation)} ({progress.table}): {done}/{total} parts{replicas}" + + def _mutation_totals(self, progress: MutationProgress) -> str: + parts = sum(self.parts_total.values()) + text = f"{_named(progress.mutation)} · {parts} {'part' if parts == 1 else 'parts'}" + if self.bytes_total and len(self.bytes_total) == len(self.parts_total): + text += f" · {_size(sum(self.bytes_total.values()))}" + if len(self.parts_total) > 1: + text += f" · {len(self.parts_total)} replicas" + return text + + +def _key(progress: Progress) -> str: + if isinstance(progress, MutationProgress): + return f"mutation {progress.table} {progress.mutation}" + return "status" + + +def _bar_row(label: str, bar: ProgressBar, stats: str) -> Table: + row = Table.grid(padding=(0, 1)) + row.add_column(no_wrap=True) + row.add_column(width=BAR_WIDTH) + row.add_column(no_wrap=True) + row.add_row(Text(f" {label}".rstrip() if label else " "), bar, Text(stats)) + return row + + +def _bar(completed: float, total: float) -> ProgressBar: + return ProgressBar(total=max(total, 1), completed=min(completed, total), width=BAR_WIDTH) + + +def _named(mutation: str) -> str: + # MergeTree ids already read "mutation_7.txt"; replicated ones are bare numbers. + return mutation if mutation.startswith("mutation") else f"mutation {mutation}" + + +def _duration(seconds: float) -> str: + seconds = max(0, int(seconds)) + hours, rest = divmod(seconds, 3600) + minutes, secs = divmod(rest, 60) + return f"{hours}:{minutes:02d}:{secs:02d}" if hours else f"{minutes}:{secs:02d}" + + +def _sizes(done: int, total: int) -> str: + unit, scale = _unit(total) + return f"{done / scale:.1f} / {total / scale:.1f} {unit}" + + +def _size(total: int) -> str: + unit, scale = _unit(total) + return f"{total / scale:.1f} {unit}" + + +def _unit(total: int) -> tuple[str, int]: + for unit, scale in (("GiB", 1 << 30), ("MiB", 1 << 20), ("KiB", 1 << 10)): + if total >= scale: + return unit, scale + return "B", 1 diff --git a/ch_migrate/waiting/errors.py b/ch_migrate/waiting/errors.py new file mode 100644 index 0000000..311798a --- /dev/null +++ b/ch_migrate/waiting/errors.py @@ -0,0 +1,37 @@ +"""Why waiting stopped short: a deadline, a failed mutation, or a deployment it can't check.""" + +from __future__ import annotations + +from dataclasses import dataclass + + +class WaitingError(RuntimeError): + """Server work is unfinished; the revision must not be completed.""" + + +class WaitTimeout(WaitingError): + """The invocation's waiting deadline expired without cancelling server work.""" + + def __init__(self, message: str, pending: str, elapsed: float) -> None: + super().__init__(message) + self.pending, self.elapsed = pending, elapsed + + +class MutationFailed(WaitingError): + """ClickHouse reports `latest_fail_reason` for a mutation this run owns or waits behind.""" + + def __init__(self, message: str, failure: FailedMutation) -> None: + super().__init__(message) + self.failure = failure + + +@dataclass(frozen=True) +class FailedMutation: + """What the operator needs: where it failed, why, and the scoped KILL statement.""" + + table: str + mutation: str + host: str + reason: str + kill: str + foreign: bool = False diff --git a/ch_migrate/waiting/mutations.py b/ch_migrate/waiting/mutations.py new file mode 100644 index 0000000..00d5763 --- /dev/null +++ b/ch_migrate/waiting/mutations.py @@ -0,0 +1,264 @@ +"""Wait until a database has no unfinished mutations on any replica that has to apply them. + +The waiter doesn't track which statement started which mutation. After each write it +records the unfinished mutations the connected server reports, then polls until each one +is done everywhere it applies: on the service for SharedMergeTree, on every replica for +ReplicatedMergeTree (a replica that hasn't pulled the mutation yet is still pending), and +where it runs for anything else. Every unfinished mutation in the database counts, so a +migration also waits for, and is stopped by, another writer's work. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass, replace + +from clickhouse_connect.driver.exceptions import DatabaseError + +from ch_migrate.waiting import reruns +from ch_migrate.waiting.display import HostParts, MutationProgress +from ch_migrate.waiting.errors import FailedMutation, MutationFailed, WaitingError +from ch_migrate.waiting.sql import sql_string + +SHARED = "shared" +# ch-migrate's own tables: version_table.py waits on the version table's DELETE itself and +# refuses a stuck one before the run starts, and the rerun markers' DELETE waits the same way. +INTERNAL_TABLES = ("alembic_version", reruns.TABLE) + + +class MutationWaiter: + """Polls `system.mutations` on the migration session; never kills or resends anything.""" + + def __init__(self, client, state, budget): + self.client, self.state, self.budget = client, state, budget + self._hosts: dict[str, tuple[str, ...] | None] = {} + self._part_sizes: dict[tuple[str, str], int] = {} + + def wait(self, before_run: bool = False) -> None: + """Return once every unfinished mutation seen now has finished where it applies.""" + self._hosts.clear() + expected = {(row.table, row.mutation) for row in self._rows(set())} + while expected: + rows = self._rows(expected) + expected |= {(row.table, row.mutation) for row in rows if not row.done} + _raise_failures(self.state.database, rows, before_run) + pending = self._pending(expected, rows) + if not pending: + self.budget.complete(f"{self.state.database} mutations") + return + self.budget.pause(_pending_text(self.state.database, pending), self._snapshot(pending)) + + def _rows(self, expected: set[tuple[str, str]]) -> list[_Row]: + """Unfinished mutations in the database, and every row for the ones already expected.""" + cluster = self.state.health_cluster + source = ( + "clusterAllReplicas({cluster:String}, system.mutations)" + if cluster + else "system.mutations" + ) + rows = self.client.query( + "SELECT materialize(hostName()), table, mutation_id, is_done, parts_to_do, " + f"latest_fail_reason, parts_to_do_names FROM {source} WHERE database = {{db:String}} " + "AND NOT has({internal:Array(String)}, table) " + "AND (NOT is_done OR has({expected:Array(String)}, concat(table, '/', mutation_id)))", + parameters={ + "db": self.state.database, + "internal": list(INTERNAL_TABLES), + "cluster": cluster, + "expected": sorted(f"{table}/{mutation}" for table, mutation in expected), + }, + settings=_cluster_settings() if cluster else None, + ).result_rows + return [self._row(values) for values in rows] + + def _row(self, values: tuple) -> _Row: + host, table, mutation, done, parts, reason, names = values + # Every replica of a SharedMergeTree table reports the same shared mutation. + if self._required_hosts(table) == (SHARED,): + host = SHARED + return _Row(host, table, mutation, bool(done), parts, reason, frozenset(names)) + + def _pending(self, expected: set[tuple[str, str]], rows: list[_Row]) -> list[_Row]: + """Unfinished rows, plus a placeholder for each replica that hasn't reported one yet.""" + pending = [row for row in rows if not row.done] + seen = {(row.host, row.table, row.mutation) for row in rows} + for table, mutation in sorted(expected): + for host in self._required_hosts(table) or (): + if host != SHARED and (host, table, mutation) not in seen: + pending.append( + _Row(host, table, mutation, False, 0, "", frozenset(), missing=True) + ) + return pending + + def _required_hosts(self, table: str) -> tuple[str, ...] | None: + """Hosts that must each report a mutation done; None means wherever it appears.""" + if table not in self._hosts: + self._hosts[table] = self._resolve_hosts(table) + return self._hosts[table] + + def _resolve_hosts(self, table: str) -> tuple[str, ...] | None: + rows = self.client.query( + "SELECT engine FROM system.tables WHERE database = {db:String} AND name = {table:String}", + parameters={"db": self.state.database, "table": table}, + ).result_rows + engine = rows[0][0] if rows else "" + if engine.startswith("Shared"): + return (SHARED,) + if engine.startswith("Replicated"): + return self._replicas(table) + return None + + def _replicas(self, table: str) -> tuple[str, ...]: + path, total = self.client.query( + "SELECT zookeeper_path, total_replicas FROM system.replicas " + "WHERE database = {db:String} AND table = {table:String}", + parameters={"db": self.state.database, "table": table}, + ).result_rows[0] + cluster = self.state.health_cluster + if not cluster: + if total > 1: + raise WaitingError( + f"{self.state.database}.{table} has {total} replicas; set `cluster` in " + "config.yaml so `up` can wait for the mutation on every one" + ) + return (self.client.command("SELECT hostName()"),) + while len(hosts := self._cluster_replicas(table, path)) < total: + self.budget.pause( + f"{self.state.database}.{table}: cluster {cluster} shows {len(hosts)}/{total} replicas" + ) + return hosts + + def _cluster_replicas(self, table: str, path: str) -> tuple[str, ...]: + """Hosts the cluster can reach that hold a replica of this table's Keeper path.""" + rows = self.client.query( + "SELECT DISTINCT materialize(hostName()) FROM clusterAllReplicas({cluster:String}, " + "system.replicas) WHERE database = {db:String} AND table = {table:String} " + "AND zookeeper_path = {path:String}", + parameters={ + "cluster": self.state.health_cluster, + "db": self.state.database, + "table": table, + "path": path, + }, + settings=_cluster_settings(), + ).result_rows + return tuple(sorted(host for (host,) in rows)) + + def _snapshot(self, pending: list[_Row]) -> MutationProgress: + """The oldest pending mutation, as the display shows it.""" + table, mutation = min(((row.table, row.mutation) for row in pending), key=_mutation_order) + rows = [row for row in pending if (row.table, row.mutation) == (table, mutation)] + # A replica that hasn't received the mutation has every part its peers report. + most = max((len(row.names) or row.parts for row in rows if not row.missing), default=0) + hosts = [ + HostParts(row.host, row.names, most if row.missing else len(row.names) or row.parts) + for row in rows + ] + if self.budget.display.wants_bytes: + hosts = self._with_sizes(table, hosts) + return MutationProgress(f"{self.state.database}.{table}", mutation, tuple(hosts)) + + def _with_sizes(self, table: str, hosts: list[HostParts]) -> list[HostParts]: + # Parts never change size, so each is read once; only new names cost a query. + wanted = {(host.host, name) for host in hosts for name in host.names} + unknown = {name for _, name in wanted - self._part_sizes.keys()} + if unknown: + try: + self._part_sizes.update(self._read_sizes(table, unknown)) + except DatabaseError as error: + self.budget.display.parts_only(_sizes_unreadable(error)) + return hosts + sizes = self._part_sizes + return [ + replace( + host, + sizes={n: sizes[(host.host, n)] for n in host.names if (host.host, n) in sizes}, + ) + for host in hosts + ] + + def _read_sizes(self, table: str, names: set[str]) -> dict[tuple[str, str], int]: + cluster = self.state.health_cluster + source = "clusterAllReplicas({cluster:String}, system.parts)" if cluster else "system.parts" + rows = self.client.query( + f"SELECT materialize(hostName()), name, bytes_on_disk FROM {source} " + "WHERE database = {db:String} AND table = {table:String} AND active " + "AND has({names:Array(String)}, name)", + parameters={ + "cluster": cluster, + "db": self.state.database, + "table": table, + "names": sorted(names), + }, + settings=_cluster_settings() if cluster else None, + ).result_rows + shared = self._required_hosts(table) == (SHARED,) + return {(SHARED if shared else host, name): size for host, name, size in rows} + + +@dataclass(frozen=True) +class _Row: + """One host's view of one mutation, or a placeholder for a replica that hasn't got it.""" + + host: str + table: str + mutation: str + done: bool + parts: int + reason: str + names: frozenset[str] + missing: bool = False + + +def _raise_failures(database: str, rows: list[_Row], before_run: bool) -> None: + for row in rows: + if row.done or not row.reason: + continue + kill = ( + f"KILL MUTATION WHERE database = {sql_string(database)} " + f"AND table = {sql_string(row.table)} AND mutation_id = {sql_string(row.mutation)}" + ) + table = f"{database}.{row.table}" + prefix = "Blocked by a failed mutation" if before_run else "Mutation failed" + raise MutationFailed( + f"{prefix}: {table} mutation {row.mutation} on {row.host}: {row.reason}\n" + f"Operator only on {row.host}: {kill}", + FailedMutation(table, row.mutation, row.host, row.reason, kill, before_run), + ) + + +def _pending_text(database: str, pending: list[_Row]) -> str: + parts = [] + for row in sorted(pending, key=lambda row: (row.table, _order(row.mutation), row.host)): + where = f"{database}.{row.table} mutation {row.mutation} host {row.host}" + parts.append( + f"{where}; not received yet" if row.missing else f"{where}; parts_to_do={row.parts}" + ) + return "; ".join(parts) + + +def _mutation_order(key: tuple[str, str]) -> tuple[int, str]: + return _order(key[1]), key[0] + + +def _order(mutation: str) -> int: + numbers = re.findall(r"\d+", mutation) + return int(numbers[-1]) if numbers else 0 + + +def _cluster_settings() -> dict: + # A replica that's down is skipped, so its missing rows read as pending, not as done. + return { + "skip_unavailable_shards": 1, + "connect_timeout_with_failover_ms": 250, + "receive_timeout": 2, + } + + +def _sizes_unreadable(error: Exception) -> str: + code = re.search(r"\(([A-Z][A-Z_]+)\)", str(error)) + why = f" ({code.group(1)})" if code else "" + return ( + f"can't read system.parts{why}. Grant SELECT ON system.parts to the migration role " + "to see bytes and a time estimate." + ) diff --git a/ch_migrate/waiting/reruns.py b/ch_migrate/waiting/reruns.py new file mode 100644 index 0000000..c6c2a26 --- /dev/null +++ b/ch_migrate/waiting/reruns.py @@ -0,0 +1,182 @@ +"""Stop a rerun of an interrupted migration step that ch-migrate can't vouch for. + +Before a step runs, `up` and `down` record that it started; once its version is recorded, +the record goes. A record still there on the next run means the step was interrupted, so +some of its statements may already have applied. The step then runs again on its own only +if every statement in it passes the idempotency gate without a waiver. Otherwise the run +stops until someone checks what the earlier attempt changed and names the revision with +`--rerun-interrupted`. +""" + +from __future__ import annotations + +import logging +from dataclasses import dataclass +from pathlib import Path +from urllib.parse import quote +from uuid import uuid4 + +from clickhouse_connect.driver.binding import quote_identifier + +from ch_migrate.idempotency import classify_idempotency, waiver_reason +from ch_migrate.statements import MigrationStatement, migration_statements +from ch_migrate.waiting.errors import WaitingError +from ch_migrate.waiting.sql import sql_string + +TABLE = "_ch_migrate_in_progress" +logger = logging.getLogger(__name__) + + +class RerunGuard: + """Records each step as it starts and refuses an unsafe rerun of an interrupted one.""" + + def __init__(self, client, state, acknowledged: frozenset[str]): + self.client, self.state, self.acknowledged = client, state, acknowledged + self.before_first_upgrade = None # set by env.py when the gate hasn't run yet + self._ready = False + + def steps(self, original): + """Wrap Alembic's step iterator: check and record each step, clear it once recorded.""" + + def iterate(heads, context): + for step in original(heads, context): + if not hasattr(step, "revision"): + yield step # Stamp steps run no migration body. + continue + direction = "upgrade" if step.is_upgrade else "downgrade" + current = Step(step.revision.revision, direction, Path(step.revision.path)) + self.begin(current) + # Alembic asks for the next step only once this one's version is recorded. + yield step + self.finish(current) + + return iterate + + def begin(self, step: Step) -> None: + if step.direction == "upgrade" and self.before_first_upgrade is not None: + check, self.before_first_upgrade = self.before_first_upgrade, None + check() + self._prepare() + if self._started(step): + unvouched = unvouched_statements(step.path, step.direction) + if unvouched and not self._acknowledged(step.revision): + raise InterruptedStep(step, unvouched) + logger.warning( + f"{step.revision[:8]} was interrupted on an earlier run; running it again" + ) + self._record(step) + + def finish(self, step: Step) -> None: + self.client.command( + f"ALTER TABLE {self._table()} DELETE WHERE revision = {sql_string(step.revision)} " + "SETTINGS mutations_sync = 2" + ) + + def _prepare(self) -> None: + if self._ready: + return + if not self.client.command(f"EXISTS TABLE {self._table()}"): + self.client.command(self._create_sql()) + if self._replicated(): + # Fetch markers another replica wrote; there are no mutations here to wait for. + self.client.command(f"SYSTEM SYNC REPLICA {self._table()} LIGHTWEIGHT") + self._ready = True + + def _started(self, step: Step) -> bool: + return bool( + self.client.command( + f"SELECT count() FROM {self._table()} " + "WHERE revision = {revision:String} AND direction = {direction:String}", + parameters={"revision": step.revision, "direction": step.direction}, + settings={"select_sequential_consistency": 1} if self._shared() else None, + ) + ) + + def _acknowledged(self, revision: str) -> bool: + return any(len(given) >= 4 and revision.startswith(given) for given in self.acknowledged) + + def _record(self, step: Step) -> None: + self.client.command( + f"INSERT INTO {self._table()} (revision, direction) " + "SELECT {revision:String}, {direction:String}", + parameters={"revision": step.revision, "direction": step.direction}, + # Replicated and Shared MergeTree drop a block identical to an earlier insert. + settings={ + "insert_deduplication_token": f"chm_started_{uuid4().hex}", + "async_insert": 0, + }, + ) + + def _create_sql(self) -> str: + engine, on_cluster = "MergeTree", "" + if self.state.database_engine == "Replicated": + engine = "ReplicatedMergeTree" + elif not self._shared() and self.state.cluster: + path = f"/clickhouse/ch_migrate/{quote(self.state.database, safe='')}/{{shard}}/{TABLE}" + engine = f"ReplicatedMergeTree({sql_string(path)}, '{{replica}}')" + on_cluster = f" ON CLUSTER {quote_identifier(self.state.on_cluster)}" + return ( + f"CREATE TABLE IF NOT EXISTS {self._table()}{on_cluster} (revision String, " + "direction LowCardinality(String), started DateTime64(3) DEFAULT now64(3)) " + f"ENGINE = {engine} ORDER BY (revision, direction)" + ) + + def _table(self) -> str: + return f"{quote_identifier(self.state.database)}.{quote_identifier(TABLE)}" + + def _shared(self) -> bool: + return self.state.database_engine.startswith("Shared") + + def _replicated(self) -> bool: + return self.state.database_engine == "Replicated" or ( + not self._shared() and bool(self.state.cluster) + ) + + +@dataclass(frozen=True) +class Step: + revision: str + direction: str # "upgrade" or "downgrade" + path: Path + + +class InterruptedStep(WaitingError): + """An interrupted step has statements that might not be safe to run again.""" + + def __init__(self, step: Step, unvouched: list[Unvouched]): + self.step, self.unvouched = step, unvouched + listed = "; ".join(f"{_where(item.statement)} ({item.why})" for item in unvouched) + super().__init__( + f"{step.revision[:8]} was interrupted on an earlier run, and ch-migrate can't vouch " + f"for running it again: {listed}. Check what the earlier run changed, then run it " + f"again with --rerun-interrupted {step.revision[:8]} " + f"(with alembic, -x rerun_interrupted={step.revision[:8]})." + ) + + +@dataclass(frozen=True) +class Unvouched: + """A statement in an interrupted step that might not be safe to run again.""" + + statement: MigrationStatement + why: str + + +def unvouched_statements(path: Path, direction: str) -> list[Unvouched]: + """Statements in this step the gate can't vouch for: waived, unreadable, or unsafe.""" + found = [] + for statement in migration_statements(path): + if statement.direction != direction: + continue + reason = waiver_reason(statement.comments) + if reason is not None: + found.append(Unvouched(statement, f"waiver: {reason}" if reason else "empty waiver")) + elif not statement.readable: + found.append(Unvouched(statement, "SQL that only exists at run time")) + elif classify_idempotency(statement.sql).status != "ok": + found.append(Unvouched(statement, "not safe to run twice")) + return found + + +def _where(statement: MigrationStatement) -> str: + return f"{statement.source}:{statement.line}" diff --git a/ch_migrate/waiting/sql.py b/ch_migrate/waiting/sql.py new file mode 100644 index 0000000..bb28485 --- /dev/null +++ b/ch_migrate/waiting/sql.py @@ -0,0 +1,36 @@ +"""The little SQL the waiter needs: which statements can start work, and string literals.""" + +from __future__ import annotations + +import re + +from ch_migrate.sql import _skip_quoted_or_comment + +# Statements that only read or change the session; everything else may start a mutation +# or distributed DDL. Waiting after a read costs one poll, so unknown forms count as writes. +_READS = frozenset(("SELECT", "SHOW", "DESCRIBE", "DESC", "EXPLAIN", "EXISTS", "SET", "USE")) +_WORD = re.compile(r"[A-Za-z]+") + + +def is_write(statement: str) -> bool: + """True unless the statement's first keyword only reads or sets session state.""" + word = _WORD.match(_first_code(statement)) + return word is None or word.group(0).upper() not in _READS + + +def sql_string(value: str) -> str: + return "'" + value.replace("\\", "\\\\").replace("'", "\\'") + "'" + + +def _first_code(sql: str) -> str: + """The statement from its first character that isn't whitespace or a comment.""" + i = 0 + while i < len(sql): + if sql[i].isspace(): + i += 1 + continue + skip_to = _skip_quoted_or_comment(sql, i) + if skip_to is None or sql[i] in "'\"`": + return sql[i:] + i = skip_to + return "" diff --git a/ch_migrate/waiting/waiter.py b/ch_migrate/waiting/waiter.py new file mode 100644 index 0000000..bc75c2e --- /dev/null +++ b/ch_migrate/waiting/waiter.py @@ -0,0 +1,68 @@ +"""Hold each migration until ClickHouse has finished the work its statements started.""" + +from __future__ import annotations + +from sqlalchemy import event + +from ch_migrate.waiting.budget import WaitBudget +from ch_migrate.waiting.mutations import MutationWaiter +from ch_migrate.waiting.sql import is_write + +# ON CLUSTER statements wait inside ClickHouse. The limit stays below clickhouse-connect's +# default 300-second read timeout, so ClickHouse answers before the client gives up. +DDL_WAIT_LIMIT = 270 + + +class MigrationWaiter: + """Waits after every write on the migration connection; reads and `SET`s pass through. + + ON CLUSTER statements return only once every host has run them; a host that fails or + runs out of time is an error. After each write it waits for the database's unfinished + mutations. + """ + + def __init__(self, connection, state, timeout: float | None = None): + self.connection = connection + self.client = connection.connection.dbapi_connection.client + self.budget = WaitBudget(timeout) + self.mutations = MutationWaiter(self.client, state, self.budget) + self._installed = False + + def install(self, config) -> None: + # `ch-migrate up` passes the display it labels with each migration's name. + self.budget.display = config.attributes.get("ch_migrate_display", self.budget.display) + # Request settings, not a session SET, so a migration's own SETs can't undo them. + # wait_end_of_query makes a DDL timeout arrive as an error, not a truncated response. + self.client.set_client_setting("distributed_ddl_output_mode", "throw") + self.client.set_client_setting("distributed_ddl_task_timeout", DDL_WAIT_LIMIT) + self.client.set_client_setting("wait_end_of_query", 1) + event.listen(self.connection, "before_cursor_execute", self._before) + event.listen(self.connection, "after_cursor_execute", self._after) + self._installed = True + + def wait(self, before_run: bool = False) -> None: + """Wait for the database's unfinished mutations, including another writer's.""" + self.mutations.wait(before_run) + + def close(self) -> None: + self.budget.finish() + if self._installed: + event.remove(self.connection, "before_cursor_execute", self._before) + event.remove(self.connection, "after_cursor_execute", self._after) + self._installed = False + + def _before(self, conn, cursor, statement, parameters, context, executemany) -> None: + if is_write(statement) and self.budget.timeout is not None: + self._limit_distributed_ddl() + + def _after(self, conn, cursor, statement, parameters, context, executemany) -> None: + if is_write(statement): + self.wait() + + def _limit_distributed_ddl(self) -> None: + # `--timeout` covers the whole run, so ClickHouse waits only for what's left of it. + self.budget.check("the next statement") + left = int(self.budget.timeout - self.budget.elapsed) + self.client.set_client_setting( + "distributed_ddl_task_timeout", max(1, min(DDL_WAIT_LIMIT, left)) + ) diff --git a/ch_migrate/waiting_ddl.py b/ch_migrate/waiting_ddl.py deleted file mode 100644 index ab6c2c9..0000000 --- a/ch_migrate/waiting_ddl.py +++ /dev/null @@ -1,112 +0,0 @@ -"""Distributed DDL receipts identified by an owned queue marker, never query text.""" - -from __future__ import annotations - -from ch_migrate.waiting_types import UnknownOutcome, WaitingError - - -class DistributedDDLWaiter: - def __init__(self, client, budget): - self.client, self.budget = client, budget - - def prepare(self, cluster: str, token: str) -> dict: - hosts = self.client.query( - "SELECT host_name, port FROM system.clusters WHERE cluster = {cluster:String} " - "ORDER BY shard_num, replica_num", - parameters={"cluster": cluster}, - ).result_rows - if not hosts: - raise WaitingError(f"Cluster {cluster!r} has no configured DDL hosts") - return { - "cluster": cluster, - "token": token, - "hosts": [list(host) for host in hosts], - "entry": None, - "done_hosts": [], - } - - def wait(self, receipt: dict, checkpoint) -> None: - expected = {_host_key(host, port) for host, port in receipt["hosts"]} - while not expected.issubset(receipt["done_hosts"]): - rows = self._rows(receipt) - entries = {row[0] for row in rows} - if not entries: - raise UnknownOutcome( - "Distributed DDL outcome unknown: queue evidence is missing or expired; statement not reissued" - ) - if len(entries) != 1: - raise UnknownOutcome( - "Distributed DDL ownership marker identifies multiple queue entries; reconcile without resubmitting" - ) - entry = next(iter(entries)) - if receipt["entry"] is not None and receipt["entry"] != entry: - raise UnknownOutcome( - "Distributed DDL queue entry changed for a recorded intent; statement not reissued" - ) - if receipt["entry"] is None: - receipt["entry"] = entry - checkpoint() - pending = self._host_states(receipt, rows, checkpoint) - if pending: - self.budget.pause(f"DDL {entry}: " + "; ".join(pending)) - self.budget.complete(f"DDL {receipt['entry']}; every required host finished") - - def tables( - self, table: tuple[str, str], cluster: str, allow_missing: bool = False - ) -> dict | None: - rows = self.client.query( - "SELECT materialize(hostName()), toString(uuid), engine, sorting_key FROM " - "clusterAllReplicas({cluster:String}, system.tables) " - "WHERE database = {db:String} AND name = {table:String}", - parameters={"cluster": cluster, "db": table[0], "table": table[1]}, - settings={ - "skip_unavailable_shards": int(allow_missing), - "connect_timeout_with_failover_ms": 250, - }, - ).result_rows - if not rows: - return None - if len({row[2] for row in rows}) != 1: - raise WaitingError( - "Distributed DDL target engines differ across hosts; reconcile them before tracked execution" - ) - return { - "database": table[0], - "table": table[1], - "cluster": cluster, - "ddl_scope": True, - "engine": rows[0][2], - "sorting_key": rows[0][3], - "initiator": rows[0][0], - "uuids": {host: uuid for host, uuid, _, _ in rows}, - } - - def _rows(self, receipt: dict) -> list: - return self.client.query( - "SELECT entry, host, port, status, exception_code, exception_text " - "FROM system.distributed_ddl_queue WHERE settings['log_comment'] = {token:String}", - parameters={"token": receipt["token"]}, - ).result_rows - - def _host_states(self, receipt: dict, rows: list, checkpoint) -> list[str]: - observed = {} - for _, host, port, status, code, message in rows: - key = _host_key(host, port) - if code: - raise WaitingError( - f"Distributed DDL {receipt['entry']} failed on {key}: code {code}: {message}" - ) - observed[key] = status - if status == "Finished" and code == 0 and key not in receipt["done_hosts"]: - receipt["done_hosts"].append(key) - checkpoint() - pending = [] - for host, port in receipt["hosts"]: - key = _host_key(host, port) - if key not in receipt["done_hosts"]: - pending.append(f"host {key}: {observed.get(key) or 'missing queue row'}") - return pending - - -def _host_key(host, port) -> str: - return f"{host}:{port}" diff --git a/ch_migrate/waiting_mutations.py b/ch_migrate/waiting_mutations.py deleted file mode 100644 index e46cdea..0000000 --- a/ch_migrate/waiting_mutations.py +++ /dev/null @@ -1,243 +0,0 @@ -"""Mutation ownership, predecessor handling, and per-replica completion evidence.""" - -from __future__ import annotations - -import re - -from ch_migrate.waiting_sql import sql_string -from ch_migrate.waiting_types import UnknownOutcome, WaitingError - - -class MutationWaiter: - def __init__(self, client, state, budget): - self.client, self.state, self.budget = client, state, budget - - def target(self, table: tuple[str, str]) -> dict | None: - rows = self.client.query( - "SELECT materialize(hostName()), toString(uuid), engine, sorting_key FROM system.tables " - "WHERE database = {db:String} AND name = {table:String}", - parameters={"db": table[0], "table": table[1]}, - ).result_rows - if not rows: - return None - host, uuid, engine, sorting_key = rows[0] - target = { - "database": table[0], - "table": table[1], - "engine": engine, - "initiator": host, - "uuids": {host: uuid}, - "cluster": None, - "sorting_key": sorting_key, - } - if engine.startswith("Shared"): - target["initiator"], target["uuids"] = "shared", {"shared": uuid} - elif engine.startswith("Replicated"): - self._replicated_target(target) - return target - - def prepare(self, target: dict, token: str) -> dict: - while True: - rows = self._rows(target) - pending = [row for row in rows if not row[3]] - self._failures(target, pending, "Blocked by foreign mutation") - if not pending: - self.budget.complete(f"{target['database']}.{target['table']} foreign predecessors") - break - self.budget.pause("foreign predecessor " + self._progress(target, pending)) - target = dict(target) - target.update(token=token, ids={}, done_hosts=[], baseline={}) - for host in target["uuids"]: - target["baseline"][host] = sorted({row[1] for row in rows if row[0] == host}) - return target - - def wait(self, receipt: dict, checkpoint) -> None: - while True: - identities = self._identities(receipt) - self._visible_hosts = set(identities) - for host, uuid in identities.items(): - if host in receipt["uuids"] and uuid != receipt["uuids"][host]: - raise UnknownOutcome( - f"Table {receipt['database']}.{receipt['table']} was replaced on {host}; statement not reissued" - ) - if receipt["initiator"] not in identities and not receipt.get("cluster"): - raise UnknownOutcome( - f"Table {receipt['database']}.{receipt['table']} disappeared; statement not reissued" - ) - rows = self._rows(receipt) - self._bind_markers(receipt, rows, checkpoint) - pending = self._pending(receipt, rows, checkpoint) - if not pending: - self.budget.complete( - f"{receipt['database']}.{receipt['table']}; every required replica finished" - ) - return - self.budget.pause("; ".join(pending)) - - def _replicated_target(self, target: dict) -> None: - path, total = self.client.query( - "SELECT zookeeper_path, total_replicas FROM system.replicas " - "WHERE database = {db:String} AND table = {table:String}", - parameters=self._parameters(target), - ).result_rows[0] - cluster = self.state.health_cluster - if not cluster: - if total > 1: - raise WaitingError( - "Replicated mutation waiting requires an environment cluster covering every replica" - ) - return - target["cluster"] = cluster - while True: - replicas = self.client.query( - "SELECT materialize(hostName()), replica_name FROM " - "clusterAllReplicas({cluster:String}, system.replicas) " - "WHERE database = {db:String} AND table = {table:String} AND zookeeper_path = {path:String}", - parameters={**self._parameters(target), "cluster": cluster, "path": path}, - settings=self._settings(), - ).result_rows - if len({replica for _, replica in replicas}) == total: - hosts = {host for host, _ in replicas} - target["uuids"] = { - host: uuid for host, uuid in self._identities(target).items() if host in hosts - } - if len(target["uuids"]) == total: - return - self.budget.pause( - f"{target['database']}.{target['table']}: cluster {cluster} exposes {len(replicas)}/{total} required replicas" - ) - - def _bind_markers(self, receipt: dict, rows: list, checkpoint) -> None: - found = False - for host in receipt["uuids"]: - if host in receipt["done_hosts"] or host in receipt["ids"]: - found = True - continue - local = [row for row in rows if row[0] == host] - markers = {row[1] for row in local if receipt["token"] in row[2]} - if not markers: - continue - if len(markers) != receipt.get("expected_markers", 1): - raise UnknownOutcome( - f"Outcome unknown for partially submitted batch on {host}; statement not reissued" - ) - found = True - barrier = max(_order(value) for value in markers) - baseline = set(receipt["baseline"].get(host, [])) - receipt["ids"][host] = sorted( - markers - | { - row[1] - for row in local - if not row[3] and row[1] not in baseline and _order(row[1]) <= barrier - } - ) - checkpoint() - if not found: - raise UnknownOutcome( - f"Outcome unknown for {receipt['database']}.{receipt['table']} token {receipt['token']}: " - "ownership evidence is missing or expired; statement not reissued; migration incomplete. " - "Stop runners and reconcile the journal with independent server evidence." - ) - - def _pending(self, receipt: dict, rows: list, checkpoint) -> list[str]: - pending = [] - for host in receipt["uuids"]: - if host in receipt["done_hosts"]: - continue - ids = receipt["ids"].get(host) - if not ids: - pending.append( - f"{receipt['database']}.{receipt['table']}: host {host} has not supplied the owned mutation" - ) - continue - local = [row for row in rows if row[0] == host and row[1] in ids] - missing = set(ids) - {row[1] for row in local} - if missing: - if host not in self._visible_hosts: - pending.append( - f"{receipt['database']}.{receipt['table']}: host {host} missing mutation evidence {sorted(missing)}" - ) - continue - raise UnknownOutcome( - f"Outcome unknown: mutation evidence {sorted(missing)} disappeared on {host}; statement not reissued" - ) - self._failures(receipt, local, "Mutation or preceding work failed") - unfinished = [row for row in local if not row[3]] - if unfinished: - pending.append(self._progress(receipt, unfinished)) - else: - receipt["done_hosts"].append(host) - checkpoint() - return pending - - def _rows(self, target: dict) -> list: - source, parameters = self._source(target, "mutations") - rows = self.client.query( - f"SELECT materialize(hostName()), mutation_id, command, is_done, parts_to_do, latest_fail_reason " - f"FROM {source} WHERE database = {{db:String}} AND table = {{table:String}}", - parameters=parameters, - settings=self._settings() if target.get("cluster") else None, - ).result_rows - if target["engine"].startswith("Shared"): - return [("shared", *row[1:]) for row in rows] - return [row for row in rows if row[0] in target["uuids"]] - - def _identities(self, target: dict) -> dict[str, str]: - source, parameters = self._source(target, "tables") - rows = self.client.query( - f"SELECT materialize(hostName()), toString(uuid) FROM {source} " - "WHERE database = {db:String} AND name = {table:String}", - parameters=parameters, - settings=self._settings() if target.get("cluster") else None, - ).result_rows - if target["engine"].startswith("Shared"): - return {"shared": rows[0][1]} if rows else {} - return dict(rows) - - def _source(self, target: dict, table: str) -> tuple[str, dict]: - parameters = self._parameters(target) - if target.get("cluster"): - parameters["cluster"] = target["cluster"] - return f"clusterAllReplicas({{cluster:String}}, system.{table})", parameters - return f"system.{table}", parameters - - def _settings(self) -> dict: - return { - "skip_unavailable_shards": 1, - "connect_timeout_with_failover_ms": 250, - "receive_timeout": 2, - } - - @staticmethod - def _parameters(target: dict) -> dict: - return {"db": target["database"], "table": target["table"]} - - @staticmethod - def _failures(target: dict, rows: list, prefix: str) -> None: - for host, mutation, _, done, _, reason in rows: - if not done and reason: - kill = ( - f"KILL MUTATION WHERE database = {sql_string(target['database'])} " - f"AND table = {sql_string(target['table'])} AND mutation_id = {sql_string(mutation)}" - ) - raise WaitingError( - f"{prefix}: {target['database']}.{target['table']} mutation {mutation} on {host}: {reason}\nOperator only on {host}: {kill}" - ) - - @staticmethod - def _progress(target: dict, rows: list) -> str: - parts = {} - for host, mutation, _, _, count, _ in rows: - parts[(host, mutation)] = max(parts.get((host, mutation), 0), count) - return "; ".join( - f"{target['database']}.{target['table']} mutation {mutation} host {host}; parts_to_do={count}" - for (host, mutation), count in sorted(parts.items()) - ) - - -def _order(mutation: str) -> int: - numbers = re.findall(r"\d+", mutation) - if len(numbers) != 1: - raise WaitingError(f"Unrecognized mutation ordering identifier: {mutation}") - return int(numbers[0]) diff --git a/ch_migrate/waiting_sql.py b/ch_migrate/waiting_sql.py deleted file mode 100644 index 0ba70f4..0000000 --- a/ch_migrate/waiting_sql.py +++ /dev/null @@ -1,235 +0,0 @@ -"""SQL identity and ownership markers without matching another writer's command.""" - -from __future__ import annotations - -import hashlib -import json -import re - -from clickhouse_connect.driver.binding import bind_query, quote_identifier - -from ch_migrate.classify import _identifier, _tokens -from ch_migrate.sql import _skip_quoted_or_comment -from ch_migrate.waiting_types import WaitingError - -_WORD = re.compile(r"[A-Za-z_][A-Za-z_0-9]*") -_MANAGED = frozenset(("mutations_sync", "alter_sync", "lightweight_deletes_sync")) - - -def bind_statement(statement, parameters, client) -> str: - bound, _ = bind_query(statement, parameters or None, client.server_tz) - return bound.decode() if isinstance(bound, bytes) else bound - - -def statement_digest(statement: str, parameters) -> str: - serialized = json.dumps( - [statement, parameters], sort_keys=True, default=str, separators=(",", ":") - ) - return hashlib.sha256(serialized.encode()).hexdigest() - - -def statement_table(statement: str, database: str) -> tuple[str, str] | None: - tokens = _tokens(statement) - words = tuple(token.upper() for token in tokens) - if words[:2] in (("ALTER", "TABLE"), ("INSERT", "INTO"), ("DELETE", "FROM")): - index = 2 - elif words[:1] == ("UPDATE",): - index = 1 - elif words[:1] in (("CREATE",), ("DROP",), ("RENAME",), ("TRUNCATE",)): - index = next( - (i + 1 for i, token in enumerate(words) if token in ("TABLE", "VIEW", "DICTIONARY")), - len(tokens), - ) - else: - return None - while index < len(tokens) and words[index] in ("IF", "NOT", "EXISTS"): - index += 1 - if index >= len(tokens): - return None - components = [_identifier(tokens[index])] - while index + 2 < len(tokens) and tokens[index + 1] == ".": - components.append(_identifier(tokens[index + 2])) - index += 2 - if any(component.startswith("{") for component in components): - raise WaitingError("Mutation tracking requires resolved database/table identifiers") - return (components[-2] if len(components) > 1 else database, components[-1]) - - -def statement_cluster(statement: str) -> str | None: - tokens = _tokens(statement) - for index in range(len(tokens) - 2): - if tuple(token.upper() for token in tokens[index : index + 2]) == ("ON", "CLUSTER"): - return tokens[index + 2].strip("'`\"") - return None - - -def is_session_or_read(statement: str) -> bool: - tokens = _tokens(statement) - if tokens and tokens[0].upper() == "WITH": - operation = next( - ( - span[0].upper() - for span in _spans(statement) - if span[3] == 0 and span[0].upper() in ("SELECT", "INSERT", "UPDATE", "DELETE") - ), - None, - ) - return operation == "SELECT" - return not tokens or tokens[0].upper() in ( - "SELECT", - "SHOW", - "DESCRIBE", - "DESC", - "EXPLAIN", - "SET", - "USE", - ) - - -def modifies_ttl(statement: str) -> bool: - tokens = _tokens(statement) - return any( - tuple(word.upper() for word in tokens[index : index + 2]) == ("MODIFY", "TTL") - for index in range(len(tokens) - 1) - ) - - -def mutation_sql(statement: str, token: str) -> str: - """Carry ownership into system.mutations and make submission asynchronous.""" - body, settings = _query_parts(statement) - words = _tokens(body) - marker = f"{sql_string(token)} = {sql_string(token)}" - overrides = {name: "0" for name in _MANAGED} - if tuple(word.upper() for word in words[:2]) == ("DELETE", "FROM"): - where = next( - (span for span in _spans(body) if span[0].upper() == "WHERE" and span[3] == 0), None - ) - if where is None: - raise WaitingError("A lightweight DELETE needs a WHERE predicate to track ownership") - body = body[: where[2]] + f" ({body[where[2]:].strip()}) AND ({marker})" - elif tuple(word.upper() for word in words[:2]) == ("ALTER", "TABLE"): - if modifies_ttl(body): - spans = list(_spans(body)) - start = next( - span[1] - for index, span in enumerate(spans[:-1]) - if span[0].upper() == "MODIFY" - and spans[index + 1][0].upper() == "TTL" - and span[3] == 0 - ) - body = body[:start].rstrip() + f" DELETE WHERE 0 AND ({marker}), " + body[start:] - overrides["materialize_ttl_after_modify"] = "0" - else: - body += f", DELETE WHERE 0 AND ({marker})" - else: - raise WaitingError("Cannot attach a mutation ownership marker to this statement") - return _with_settings(body, settings, overrides) - - -def query_settings(statement: str, overrides: dict[str, str]) -> str: - body, settings = _query_parts(statement) - return _with_settings(body, settings, overrides) - - -def remove_query_settings(statement: str, names: set[str]) -> str: - body, settings = _query_parts(statement) - retained = [ - item for item in settings if _identifier(item.split("=", 1)[0].strip()).lower() not in names - ] - return body + (" SETTINGS " + ", ".join(retained) if retained else "") - - -def query_setting(statement: str, name: str) -> str | None: - _, settings = _query_parts(statement) - for setting in reversed(settings): - key, _, value = setting.partition("=") - if _identifier(key.strip()).lower() == name: - return value.strip().strip("'") - return None - - -def sql_string(value: str) -> str: - return "'" + value.replace("\\", "\\\\").replace("'", "\\'") + "'" - - -def qualified_table(table: tuple[str, str]) -> str: - return ".".join(quote_identifier(part) for part in table) - - -def _query_parts(statement: str) -> tuple[str, list[str]]: - statement = _without_comments(statement) - spans = list(_spans(statement)) - if not spans: - raise WaitingError("Empty statement cannot be tracked") - sql = statement[: spans[-1][2]].rstrip() - candidates = [ - span - for index, span in enumerate(spans[:-2]) - if span[0].upper() == "SETTINGS" - and span[3] == 0 - and _WORD.fullmatch(_identifier(spans[index + 1][0])) - and spans[index + 2][0] == "=" - ] - if not candidates: - return sql, [] - settings = candidates[-1] - return sql[: settings[1]].rstrip(), _split_settings(sql[settings[2] :]) - - -def _split_settings(sql: str) -> list[str]: - starts = [span[1] for span in _spans(sql) if span[0] == "," and span[3] == 0] - result, previous = [], 0 - for index in starts: - result.append(sql[previous:index].strip()) - previous = index + 1 - if sql[previous:].strip(): - result.append(sql[previous:].strip()) - return result - - -def _with_settings(body: str, settings: list[str], overrides: dict[str, str]) -> str: - retained = [ - item - for item in settings - if _identifier(item.split("=", 1)[0].strip()).lower() not in overrides - ] - retained.extend(f"{name} = {value}" for name, value in sorted(overrides.items())) - return body + " SETTINGS " + ", ".join(retained) - - -def _without_comments(sql: str) -> str: - chunks, index = [], 0 - while index < len(sql): - end = _skip_quoted_or_comment(sql, index) - if end is None: - chunks.append(sql[index]) - index += 1 - else: - chunks.append(sql[index:end] if sql[index] in "'\"`$" else " ") - index = end - return "".join(chunks) - - -def _spans(sql: str): - """Yield significant tokens with offsets/depth, retaining quoted SQL verbatim.""" - index = depth = 0 - while index < len(sql): - end = _skip_quoted_or_comment(sql, index) - if end is not None: - if sql[index] in "'\"`$": - yield sql[index:end], index, end, depth - index = end - continue - match = _WORD.match(sql, index) - if match: - yield match.group(), index, match.end(), depth - index = match.end() - continue - char = sql[index] - if char in ")]": - depth -= 1 - if not char.isspace() and char != ";": - yield char, index, index + 1, depth - if char in "([": - depth += 1 - index += 1 diff --git a/ch_migrate/waiting_store.py b/ch_migrate/waiting_store.py deleted file mode 100644 index d746c0b..0000000 --- a/ch_migrate/waiting_store.py +++ /dev/null @@ -1,185 +0,0 @@ -"""Append-only waiting receipts, with a fresh generation after each downgrade.""" - -from __future__ import annotations - -import json -from dataclasses import dataclass - -from clickhouse_connect.cc_sqlalchemy.datatypes.sqltypes import UInt32, UInt64 -from sqlalchemy import Column, MetaData, String, Table -from sqlalchemy.schema import CreateTable - -from ch_migrate.version_table import migration_state_engine -from ch_migrate.waiting_sql import qualified_table, query_settings -from ch_migrate.waiting_types import UnknownOutcome - -JOURNAL_TABLE = "_ch_migrate_journal" - - -@dataclass(frozen=True) -class StepKey: - revision: str - generation: int - position: int - - -class WaitingJournal: - """One ordered writer per database; readers synchronize before resuming.""" - - def __init__(self, connection, state, budget): - self.connection, self.state, self.budget = connection, state, budget - self.client = connection.connection.dbapi_connection.client - self.table = qualified_table((state.database, JOURNAL_TABLE)) - self._sequences: dict[StepKey, int] = {} - self._ready = False - - def exists(self) -> bool: - return bool( - self.client.query( - "SELECT count() FROM system.tables WHERE database = {db:String} AND name = {table:String}", - parameters={"db": self.state.database, "table": JOURNAL_TABLE}, - ).result_rows[0][0] - ) - - def synchronize(self) -> None: - if not self.exists(): - return - engine = self.client.query( - "SELECT engine FROM system.tables WHERE database = {db:String} AND name = {table:String}", - parameters={"db": self.state.database, "table": JOURNAL_TABLE}, - ).result_rows[0][0] - if engine.startswith(("Replicated", "Shared")): - self.client.command(f"SYSTEM SYNC REPLICA {self.table} LIGHTWEIGHT") - - def begin(self, revision: str, fingerprint: str) -> int: - self.ensure() - generation = self._generation(revision) - if generation == 0: - generation = 1 - self.write(StepKey(revision, generation, 0), {"fingerprint": fingerprint}) - else: - control = self.read(StepKey(revision, generation, 0)) - if control is None: - raise UnknownOutcome( - f"Journal generation for {revision} has no control record; reconcile before resuming" - ) - return generation - - def invalidate(self, revision: str, fingerprint: str) -> None: - if self.exists(): - self.write( - StepKey(revision, self._generation(revision) + 1, 0), {"fingerprint": fingerprint} - ) - - def records(self, revision: str, generation: int) -> list[tuple[StepKey, dict]]: - rows = self.client.query( - f"SELECT position, max(sequence), argMax(payload, sequence) FROM {self.table} " - "WHERE revision = {revision:String} AND generation = {generation:UInt64} AND position > 0 " - "GROUP BY position ORDER BY position", - parameters={"revision": revision, "generation": generation}, - ).result_rows - result = [] - for position, sequence, payload in rows: - key = StepKey(revision, generation, position) - self._sequences[key] = sequence - result.append((key, json.loads(payload))) - return result - - def read(self, key: StepKey) -> dict | None: - rows = self.client.query( - f"SELECT sequence, payload FROM {self.table} WHERE revision = {{revision:String}} " - "AND generation = {generation:UInt64} AND position = {position:UInt32} " - "ORDER BY sequence DESC LIMIT 2", - parameters=self._parameters(key), - ).result_rows - if not rows: - self._sequences[key] = 0 - return None - if len(rows) == 2 and rows[0][0] == rows[1][0] and rows[0][1] != rows[1][1]: - raise UnknownOutcome( - f"Conflicting journal writers for {key.revision}:{key.position}; serialize runners and reconcile" - ) - self._sequences[key] = rows[0][0] - return json.loads(rows[0][1]) - - def write(self, key: StepKey, payload: dict) -> None: - if key not in self._sequences: - self.read(key) - sequence = self._sequences[key] + 1 - self.client.insert( - f"{self.state.database}.{JOURNAL_TABLE}", - [ - [ - key.revision, - key.generation, - key.position, - sequence, - json.dumps(payload, sort_keys=True), - ] - ], - column_names=["revision", "generation", "position", "sequence", "payload"], - settings={"async_insert": 0}, - ) - self._sequences[key] = sequence - - def unfinished_scope(self, revision: str) -> bool: - if not self.exists(): - return False - generation = self._generation(revision) - return any(record["phase"] != "done" for _, record in self.records(revision, generation)) - - def unfinished_versions(self) -> list[tuple[StepKey, dict]]: - if not self.exists(): - return [] - self.synchronize() - rows = self.client.query( - f"SELECT revision, generation, position, max(sequence), argMax(payload, sequence) AS latest " - f"FROM {self.table} WHERE position > 0 AND (revision, generation) IN " - f"(SELECT revision, max(generation) FROM {self.table} GROUP BY revision) " - "GROUP BY revision, generation, position HAVING JSONExtractString(latest, 'kind') = 'version' " - "AND JSONExtractString(latest, 'phase') != 'done' ORDER BY revision, position" - ).result_rows - result = [] - for revision, generation, position, sequence, payload in rows: - key = StepKey(revision, generation, position) - self._sequences[key] = sequence - result.append((key, json.loads(payload))) - return result - - def ensure(self) -> None: - if self._ready: - return - if self.exists(): - self.synchronize() - self._ready = True - return - table = Table( - JOURNAL_TABLE, - MetaData(), - Column("revision", String, nullable=False), - Column("generation", UInt64, nullable=False), - Column("position", UInt32, nullable=False), - Column("sequence", UInt64, nullable=False), - Column("payload", String, nullable=False), - migration_state_engine( - self.state, JOURNAL_TABLE, "(revision, generation, position, sequence)" - ), - schema=self.state.database, - info={"ch_migrate_on_cluster": self.state.on_cluster}, - ) - ddl = str(CreateTable(table, if_not_exists=True).compile(dialect=self.connection.dialect)) - self.client.command(query_settings(ddl, {"distributed_ddl_task_timeout": "0"})) - while not self.exists(): - self.budget.pause(f"journal {self.table} metadata is not available on this host") - self.synchronize() - self._ready = True - - def _generation(self, revision: str) -> int: - return self.client.query( - f"SELECT max(generation) FROM {self.table} WHERE revision = {{revision:String}}", - parameters={"revision": revision}, - ).result_rows[0][0] - - @staticmethod - def _parameters(key: StepKey) -> dict: - return {"revision": key.revision, "generation": key.generation, "position": key.position} diff --git a/ch_migrate/waiting_types.py b/ch_migrate/waiting_types.py deleted file mode 100644 index 47491c6..0000000 --- a/ch_migrate/waiting_types.py +++ /dev/null @@ -1,73 +0,0 @@ -"""Waiting errors, invocation deadline, and bounded progress output.""" - -from __future__ import annotations - -import sys -import time -from dataclasses import dataclass, field - - -class WaitingError(RuntimeError): - """Server work is unfinished; the revision must not be completed.""" - - -class UnknownOutcome(WaitingError): - """Available evidence cannot justify replay or completion.""" - - -class WaitTimeout(WaitingError): - """The invocation's waiting deadline expired without cancelling server work.""" - - -@dataclass -class WaitBudget: - timeout: float | None = None - started: float = field(default_factory=time.monotonic) - poll_interval: float = 0.2 - _last_line: str = "" - _last_printed: float = 0 - _live: bool = False - - @property - def elapsed(self) -> float: - return time.monotonic() - self.started - - def check(self, pending: str) -> None: - if self.timeout is not None and self.elapsed >= self.timeout: - self.finish() - raise WaitTimeout( - f"Timed out after {self.elapsed:.1f}s; server work continues; " - f"migration incomplete. Pending: {pending}" - ) - - def pause(self, pending: str) -> None: - self.check(pending) - self.progress(pending) - remaining = self.poll_interval - if self.timeout is not None: - remaining = min(remaining, max(0, self.timeout - self.elapsed)) - time.sleep(remaining) - self.check(pending) - - def progress(self, pending: str) -> None: - now = time.monotonic() - tty = sys.stderr.isatty() - if pending != self._last_line or now - self._last_printed >= (0.5 if tty else 5): - line = f"Waiting: {pending}; {self.elapsed:.1f}s elapsed" - if tty: - print("\r\033[K" + line, end="", file=sys.stderr, flush=True) - self._live = True - else: - print(line, file=sys.stderr, flush=True) - self._last_printed, self._last_line = now, pending - - def complete(self, message: str) -> None: - self.finish() - if self._last_line: - print(f"Finished: {message}; {self.elapsed:.1f}s elapsed", file=sys.stderr, flush=True) - self._last_line = "" - - def finish(self) -> None: - if self._live: - print(file=sys.stderr, flush=True) - self._live = False diff --git a/ch_migrate/waiting_versions.py b/ch_migrate/waiting_versions.py deleted file mode 100644 index 78fcca8..0000000 --- a/ch_migrate/waiting_versions.py +++ /dev/null @@ -1,163 +0,0 @@ -"""Resume insert-first version bookkeeping without invoking a revision twice.""" - -from __future__ import annotations - -import ast -from types import SimpleNamespace -from uuid import uuid4 - -from sqlalchemy import text -from sqlalchemy.sql.dml import Delete, Insert, Update - -from ch_migrate.waiting_sql import ( - mutation_sql, - qualified_table, - query_settings, - sql_string, - statement_digest, -) -from ch_migrate.waiting_types import UnknownOutcome, WaitingError - - -class VersionWrites: - def __init__(self, owner): - self.owner = owner - - def execute(self, construct, impl): - steps = _version_steps(construct, impl) - payload = { - "kind": "version", - "phase": "intent", - "steps": steps, - "digest": statement_digest(str(construct), [step["sql"] for step in steps]), - } - key, record, _ = self.owner.reserve(payload) - self.continue_record(key, record) - return SimpleNamespace(rowcount=1) - - def resume(self) -> None: - for key, record in self.owner.journal.unfinished_versions(): - self.owner.validate_revision(key.revision, record["fingerprint"]) - print( - f"Resuming version bookkeeping for {key.revision}", file=self.owner.progress_stream - ) - self.continue_record(key, record) - - def continue_record(self, key, record: dict) -> None: - if record["phase"] == "done": - return - try: - for step in record["steps"]: - if step.get("done"): - continue - - def checkpoint() -> None: - self.owner.journal.write(key, record) - - if step["kind"] == "insert": - self._insert(step, checkpoint) - else: - self._delete(step, checkpoint) - step["done"] = True - checkpoint() - record["phase"] = "done" - self.owner.journal.write(key, record) - except UnknownOutcome as error: - raise self.owner.unknown_outcome(key, record, error) from error - - def _insert(self, step: dict, checkpoint) -> None: - table = qualified_table((self.owner.state.database, "alembic_version")) - count = self.owner.client.query( - f"SELECT count() FROM {table} WHERE version_num = {{version:String}}", - parameters={"version": step["value"]}, - ).result_rows[0][0] - if step.get("prepared"): - if count == 1: - return # The exact version-row effect proves the internal INSERT succeeded. - raise UnknownOutcome( - f"Outcome unknown for version INSERT {step['value']}: found {count} rows; " - "statement not reissued; reconcile version state before resuming" - ) - if count: - raise WaitingError( - f"Version {step['value']} already exists before its INSERT; reconcile overlapping heads" - ) - step["prepared"] = True - checkpoint() - self.owner.client.command( - step["sql"], settings={"async_insert": 0, "wait_for_async_insert": 1} - ) - - def _delete(self, step: dict, checkpoint) -> None: - if not step.get("prepared"): - target = self.owner.mutations.target((self.owner.state.database, "alembic_version")) - if target is None: - raise UnknownOutcome("Version table disappeared before deleting the previous head") - token = "chm_version_" + uuid4().hex - step["receipt"] = self.owner.mutations.prepare(target, token) - step["prepared"] = True - checkpoint() - with self.owner.internal(): - self.owner.connection.exec_driver_sql(mutation_sql(step["sql"], token)) - self.owner.mutations.wait(step["receipt"], checkpoint) - - -def write_unjournaled(construct, impl): - """Apply a version INSERT/UPDATE outside a tracked upgrade, such as a downgrade.""" - for step in _version_steps(construct, impl): - sql = step["sql"] - if step["kind"] == "delete": - sql = query_settings(sql, {"mutations_sync": "2"}) - impl._exec(text(sql)) - return SimpleNamespace(rowcount=1) - - -def _version_steps(construct, impl) -> list[dict]: - if isinstance(construct, Insert): - value = _version_value(construct) - return [ - {"kind": "insert", "value": value, "sql": _insert_sql(impl._version_table_name, value)} - ] - if isinstance(construct, Update): - value = _version_value(construct) - return [ - { - "kind": "insert", - "value": value, - "sql": _insert_sql(impl._version_table_name, value), - }, - { - "kind": "delete", - "sql": f"ALTER TABLE {impl._version_table_name} DELETE WHERE {impl._compile_version_where(construct)}", - }, - ] - if isinstance(construct, Delete): - return [ - { - "kind": "delete", - "sql": f"ALTER TABLE {impl._version_table_name} DELETE WHERE {impl._compile_version_where(construct)}", - } - ] - raise WaitingError("Unsupported version-table transition") - - -def _insert_sql(table: str, value: str) -> str: - # Replicated and Shared MergeTree silently drop a block identical to an earlier insert - # (DRY-1447), so re-inserting a previous head after `down` left no version row at all. - # A token unique to this write keeps every intentional version row. - token = sql_string("chm_version_" + uuid4().hex) - return ( - f"INSERT INTO {table} (version_num) SETTINGS insert_deduplication_token = {token}, " - f"async_insert = 0 VALUES ({sql_string(value)})" - ) - - -def _version_value(construct) -> str: - values = construct._values - expression = next(iter(values.values())) if values else None - value = getattr(expression, "value", None) - if value is None and getattr(expression, "is_literal", False): - value = ast.literal_eval(str(expression)) - if not isinstance(value, str): - raise WaitingError("Version-table INSERT/UPDATE must contain a literal revision identifier") - return value diff --git a/docs/design/2026-10-02-ch-migrate-1.0.md b/docs/design/2026-10-02-ch-migrate-1.0.md index 04105de..9e5cb67 100644 --- a/docs/design/2026-10-02-ch-migrate-1.0.md +++ b/docs/design/2026-10-02-ch-migrate-1.0.md @@ -61,10 +61,10 @@ asserted. ### 1. One name, installable from PyPI -- `pip install ch-migrate-cli` / `uv tool install ch-migrate-cli` installs the CLI +- `pip install clickhouse-alembic` / `uv tool install clickhouse-alembic` installs the CLI `ch-migrate`. (PyPI rejects `ch-migrate` as too similar to an existing project, `chmigrate`.) -- The primary import package is `ch_migrate`. Existing migration files using - `clickhouse_alembic` keep working via the deprecated alias during this port. +- The import package stays `clickhouse_alembic`, so existing migration files + (`from clickhouse_alembic import get_db, read_sql`) keep working unchanged. - `ch-migrate --version` reports the released version. - CI runs the test suite on every supported Python version on every push (see Compatibility). Pushing a `vX.Y.Z` tag publishes that version through PyPI Trusted Publishing and creates the @@ -100,9 +100,8 @@ both; the README quick start contains no Python. loudly, not quietly drop its settings. - The version table cannot cause a migration to run twice. The version advances by inserting the new row and then deleting the old one, waiting for the delete. Interrupted - bookkeeping is never permission to repeat the migration: without ownership - evidence it is refused; with capability 5's journal it resumes the same - bookkeeping. The spike reproduced duplicate execution with the old asynchronous + bookkeeping is never permission to repeat the migration: overlapping heads are + refused. The spike reproduced duplicate execution with the old asynchronous `ALTER … UPDATE` approach. - The version table is created to suit the deployment: shared on ClickHouse Cloud, replicated in a replicated database or on a configured cluster, plain on a single server. @@ -113,8 +112,7 @@ both; the README quick start contains no Python. **Accept:** an integration test against a real ClickHouse server shows a `SET` in one statement taking effect in the next; an existing 0.x project upgrades with `upgrade-env` and runs `up` without editing migration files; with the version table's delete held back, -a second `up` cannot re-run the migration. It refuses unowned overlapping state, -or reattaches to proven owned bookkeeping under capability 5. +a second `up` cannot re-run the migration. It refuses the overlapping state. ### 4. Re-runnable migrations @@ -145,21 +143,26 @@ again, and reaches the same schema as a clean run. `up` does not record a migration as applied until ClickHouse has finished the work the migration started. -- After each statement that creates a mutation, `up` polls `system.mutations` until the mutation - finishes, showing progress. It does not rely on `mutations_sync`, which holds one HTTP request - open for the whole rewrite and loses the outcome if the connection drops. -- For `ON CLUSTER` DDL, `up` waits on the distributed DDL queue rather than trusting the - client-side timeout. +- After each statement that can write, `up` polls `system.mutations` until the project database + has no unfinished mutation on any replica that has to apply it, showing progress. It does not + rely on `mutations_sync`, which holds one HTTP request open for the whole rewrite. +- For `ON CLUSTER` DDL, ClickHouse waits for every host (`distributed_ddl_output_mode = 'throw'`) + within a limit below the client's read timeout, so a failing or unfinished host is an error, + not a quiet client-side timeout. - Waiting is on by default. There is no default timeout; `--timeout` sets one. - Whenever `up` stops waiting (a timeout, a killed CI job, a dropped connection), the outcome is the same: the migration is not recorded as applied, ClickHouse keeps doing the work, and the - next `up` finds the unfinished mutation and waits on it instead of issuing the statement again. - Issuing `UPDATE … SET n = n + 1` twice applies it twice. -- **Recovery exception approved on 2026-10-04:** if an interrupted statement's - outcome cannot be proven (for example, ownership evidence has expired), `up` - exits nonzero without reissuing the statement or marking the migration complete. - It identifies the unresolved work and requires operator reconciliation. Known - unfinished work still reattaches automatically. + next `up` waits for that work to finish, then runs the migration again. That relies on + capability 4: every statement must be safe to repeat, or carry a written waiver. A + migration noted as started but not recorded reruns on its own only if every statement + passes the gate without a waiver; otherwise `up` stops and names the statements until + someone checks and passes `--rerun-interrupted REV` (Dan, 2026-10-27, DRY-1609). +- **Design change, 2026-10-27 (Dan):** the design approved on 2026-10-04 recorded which work + each statement started in a journal, so a rerun would never send a statement twice. It took + about 1,300 lines, duplicated what capability 4 already guarantees, and left a killed or + rejected mutation unrecoverable without editing the journal by hand (DRY-1604). 0.7 waits by + observation instead; the [waiting design](2026-10-02-waiting.md) keeps the journal design and + its experiments for the record. - A failing mutation does not end on its own: ClickHouse retries it and holds back every later mutation on the same table. `up` reads `latest_fail_reason`, fails the migration, and prints the `KILL MUTATION` statement for the operator. ch-migrate never kills a mutation itself. @@ -222,25 +225,11 @@ primary key or engine. Writers keep running throughout. The mechanism and its ev - After the swap, it reports any entries ClickHouse rejected. - **Writers that wait for acknowledgement** (the default) may see a transient error at the swap, and should retry. -- **Detected, not prevented (Dan, 2026-10-13):** an insert that resolves the table just before - `EXCHANGE` and builds its view chain just after it writes only to the old table. After the - swap, `rebuild_table` finds such rows through `system.part_log` and stops for the operator, - keeping its lock, until they are copied and the lock is released. Evidence and trial rates: - `2026-10-13-rebuild-on-cloud.md`, "Added after approval". - **Out of scope for 1.0:** exactly-once rebuilds with writers running, because they need merges stopped on the live table for the whole copy. -The steps. Each is checked before it runs, so the operation can resume from persisted -checkpoints (capability 4), and a per-table lock prevents two runs at once. - -**Recovery decision (Dan, 2026-10-04): no automatic expired-lock takeover.** -A paused owner can otherwise issue a late `EXCHANGE` after a successor has swapped, -reversing the swap. An existing lock, even with an expired heartbeat, therefore stops -the new invocation with recovery instructions. An operator must establish that the -previous runner cannot resume and reconcile its in-flight work before explicitly -releasing the lock. The next invocation then acquires ownership and resumes the -persisted checkpoints, cancelling orphan copies before resetting incomplete stages. -There is no CLI bypass. Normal mutation reattachment is unchanged. +The steps. Each is checked before it runs, so the operation is safe to re-run (capability 4), and +a per-table lock prevents two runs at once: 1. Create the replacement table, and a materialized view that copies new inserts from the old table into it. @@ -255,12 +244,6 @@ There is no CLI bypass. Normal mutation reattachment is unchanged. Dropping it waits for inserts already in flight, so it acts as a barrier. Materialized views follow the table name across the swap, so dependent views need no re-pointing. -On ClickHouse Cloud (decided 2026-10-13; protocol in `2026-10-13-rebuild-on-cloud.md`), helpers -prove ownership with an owner token in their `COMMENT` and a read-back UUID, because Cloud -rejects chosen UUIDs. Step 5 relies on `EXCHANGE` draining the old async-insert queue, since -Cloud forbids the flush. The migration must declare `writers_on_this_service_only=True`, -because ch-migrate cannot see inserts made through other compute services. - **Supported:** a single server, ClickHouse Cloud, and replicated tables. Refused with a clear message, before anything is created: - a sharded table (a `Distributed` table over per-shard tables), because the swap cannot be @@ -271,10 +254,8 @@ message, before anything is created: **Accept:** an integration test inserts continuously during a rebuild that changes the sorting key; after the swap, every acknowledged row is present and every duplicate was committed inside -the documented window. After an interrupted rebuild, a rerun refuses while the old -lock remains. Once the operator establishes that the previous runner cannot resume, -reconciles its work and releases the lock, `up` resumes from the last copied partition -and completes. +the documented window; stopping the rebuild partway and running `up` again resumes from the last +copied partition and completes. ## What 1.0 is not diff --git a/docs/design/2026-10-02-rebuild-spike.md b/docs/design/2026-10-02-rebuild-spike.md index bc8bfa2..b835050 100644 --- a/docs/design/2026-10-02-rebuild-spike.md +++ b/docs/design/2026-10-02-rebuild-spike.md @@ -87,35 +87,7 @@ Never drop `t_dual` before `EXCHANGE`: rows inserted in between would be lost. **Gotcha:** `CREATE TABLE t_snap AS t` fails with `REPLICA_ALREADY_EXISTS` when `t` has an explicit Keeper path. Generate DDL with a distinct path for each helper table. -## C. ClickHouse Cloud - -### Cloud results (2026-10-13) - -These experiments ran against the DRY Cloud test service (ClickHouse 26.6.1.2326, -two replicas, Shared database) with only the host, the database names (`it_spike_*`) -and cluster-wide reads of per-node system tables changed. Redacted output is in -`spikes/2026-10-02-rebuild/cloud/`. - -| Experiment | Cloud | Same as self-hosted? | -|---|---|---| -| A: views across `EXCHANGE` | Source and `TO` views follow the name; `dependencies_table` unchanged; the sink got each id once. | Same | -| B1: `ATTACH PARTITION FROM` | 5M rows attached and visible on both replicas. ORDER BY or partition-key mismatch is rejected (36); a type change or extra column is rejected (122); column order, skip index, granularity, codec, TTL and MergeTree → ReplacingMergeTree are accepted. The script's preload setting `max_partitions_per_insert_block` is constraint-locked (452), so the counts come from a re-run without it. | Same | -| B2: unpartitioned, `MOVE PARTITION` | `ATTACH` with `tuple()`, `ID 'all'` and `ALL` works; `MOVE` with `tuple()` and `ID 'all'` works; `MOVE PARTITION ALL` is rejected (344); a different ORDER BY on `MOVE` is rejected (36). | Same | -| B3: replicated helpers | An explicit `ReplicatedMergeTree('', '')` is rejected (80) in a Shared database. Between SharedMergeTree tables, `ATTACH` and `MOVE` give identical counts on both replicas. | Different engine DDL; same transfer behaviour | -| D4: insert in flight across `EXCHANGE` + `DROP t_dual SYNC` | All 15,000 ids reached the new table, the old table and the downstream sink. `EXCHANGE` took 5–62 ms; `DROP ... SYNC` waited 13 s for the insert. | Same | -| D5: async inserts at the swap | Queued `wait=0` rows were acknowledged and lost; queued `wait=1` inserts got 741. `SYSTEM FLUSH ASYNC INSERT QUEUE` is denied (497, needs `ON *.*`). | Same loss mechanism; the flush is unavailable | -| E: end to end with writers | First run: step 5 stopped at the denied queue flush. Re-run without the flush (`e_noflush_plain.out`): 6M preloaded rows with none missing or duplicated; sync and `wait=1` writers lost nothing; 200 acknowledged `wait=0` rows were lost at the swap; 19,020 duplicate ids all inside the 4.4 s window and none outside; the dependent view's sink got every id exactly once. The kill variant (`e_noflush_kill*.out`) failed in the prototype, not on Cloud: it sent a node-local `KILL QUERY` without waiting, and the server-side copy was still running 2 s later. | Same, once the engine kills `ON CLUSTER` and waits for the copy to stop | - -**Outcome:** the physical mechanics (`EXCHANGE`, `ATTACH`/`MOVE PARTITION`, view -name-following, the in-flight insert barrier) behave as they do on self-hosted. -Cloud refuses explicit table UUIDs (36; `database_replicated_allow_explicit_uuid` -is read-only), `SYSTEM STOP MERGES` and `SYSTEM FLUSH ASYNC INSERT QUEUE`. Dan -approved the protocol in `2026-10-13-rebuild-on-cloud.md`: owner-token comments -with read-back UUIDs, plain TTL-free helpers instead of STOP MERGES, no forced -flush (the async report becomes a lower bound), and a required -`writers_on_this_service_only=True` declaration with a grace pause (DRY-1445). - -### Before the Cloud run (2026-10-02) +## C. ClickHouse Cloud: unverified - No ClickHouse doc says whether SharedMergeTree supports `ATTACH PARTITION FROM` or `MOVE PARTITION TO TABLE`. @@ -142,24 +114,6 @@ flush (the async report becomes a lower bound), and a required - **Before promising Cloud:** run the A, B, D4, D5 and E scripts against a Cloud service. Only the host changes. -**Source/changelog cross-check during DRY-1406 reproduction:** the public -[`MergeTreeData::checkStructureAndGetMergeTreeData` implementation at -`5b43330791c5ff`](https://github.com/ClickHouse/ClickHouse/blob/5b43330791c5ff/src/Storages/MergeTree/MergeTreeData.cpp) -checks physical columns, sorting/partition/primary keys, format version, indexes, -and projections for partition transfers. That is evidence for the common -MergeTree path, not proof of the Cloud service's behavior. The -[2025 changelog](https://clickhouse.com/docs/resources/changelogs/oss/2025) -records support for partition operations on `plain_rewritable` disks -([#77406](https://github.com/ClickHouse/ClickHouse/pull/77406)) and transfer between -different `plain_rewritable` disks -([#79566](https://github.com/ClickHouse/ClickHouse/pull/79566)). Storage-disk support -is not a SharedMergeTree acceptance result. The -[partition reference](https://clickhouse.com/docs/reference/statements/alter/partition) -requires matching structures, keys, and storage policies, with same replicated -versus non-replicated engine family for MOVE. The -[cross-database Replicated queue issue](https://github.com/ClickHouse/ClickHouse/issues/116383) -remains open. None of these sources replaces the authorized Cloud A/B/D/E runs. - ## D. Insert races **D1. An INSERT that started before `CREATE MATERIALIZED VIEW` does not go through the view.** @@ -286,19 +240,13 @@ and 0 duplicated preload rows. **A per-table lock is required.** The first prototype run killed only the `uv` wrapper. Two rebuilds of the same table then ran at once and left the tables in a bad state. -**Amended decision (Dan, 2026-10-04):** -- `CREATE TABLE ._chm_rebuild_lock_` remains the atomic acquisition operation, - with an owner and diagnostic heartbeat. The holder releases its own lock with DROP. -- An existing lock always refuses a newcomer. Heartbeat expiry is not permission - to take over: a paused predecessor can still submit a late `EXCHANGE`. -- An owned-container counterexample took live rows `[1, 2]` back to `[1]` after the - stale predecessor's exchange, while the lock still belonged to the successor. -- The operator must establish that the previous runner cannot resume, reconcile - its in-flight work, and explicitly release the lock. Only then may another runner - acquire ownership and resume the recorded checkpoints, using the KILL-before-stage-reset - procedure above. There is no automatic takeover or CLI bypass. -- This amendment changes expired-owner recovery, not normal mutation reattachment - or the no-acknowledged-row-loss guarantee. +Proposal (untested): +- `CREATE TABLE ._chm_rebuild_lock_ (owner String, heartbeat DateTime)` acts as the mutex, + because CREATE fails if the table already exists. +- The holder updates its heartbeat. +- A newcomer takes over only after the heartbeat expires, and only after running the KILL step + for the stale owner's query_ids. +- The lock is released with DROP. ## Other changes to the mechanism @@ -327,81 +275,3 @@ rebuilds of the same table then ran at once and left the tables in a bad state. `deduplicate_blocks_in_dependent_materialized_views`, and `EXCHANGE` in a Replicated database. - The REVOKE-based block on async inserts, and the CREATE TABLE lock. - Scale beyond 6 million rows. - -## Owned-container reproduction (DRY-1406) - -Reproduced on 2026-10-02 with ClickHouse 26.3.39.7 and Python 3.12. The historical -wrappers name `spike-rebuild-ch`, use a fixed port, and one uses broad `pkill`. -Do not run those wrappers against an existing machine setup. Use the explicit -probe from the repository root: - -```bash -env -u CH_MIGRATE_IT_URL uv run --locked --python 3.12 pytest -q -s \ - -m integration tests/integration/rebuild_spike_probe.py -``` - -The fixture starts and removes its own container. A scoped adapter translates -only the archived `docker exec spike-rebuild-ch clickhouse-client ...` invocation -to that fixture's loopback endpoint. The D scripts' SQL is unchanged; no Docker -command from those scripts runs. Credentials remain in the temporary project's -environment. E imports `e2e.py`, changes only its endpoint/database/stop-file -bindings and adds query IDs and server-time observations. The original rebuild -steps, 6-million-row preload and three writers remain in use. All writer threads -are stopped and joined; all clients are closed. - -Corrections required to make the archived evidence reproducible: - -- A's final display query ordered a UNION branch by an alias defined only in the - other branch (code 47 on this server). A parenthesized UNION now defines `tbl` - in both branches; the CREATE/INSERT/EXCHANGE operations did not change. -- A clickhouse-client output format is not an INSERT input format. The HTTP - adapter applies formatting only to result-producing statements; - `raw_query(fmt=...)` appends `FORMAT` to SQL, which otherwise corrupts VALUES - payloads and makes SYSTEM commands fail. -- E's old verifier uses client send/ack timestamps. One preliminary run reported - 600 fire-and-forget IDs outside the client-time window because enqueue happened - before the view and flush happened afterward. The accepted window is in server - time. The reproduction attributes every duplicate batch with query IDs: - synchronous execution intervals from `system.query_log`, and asynchronous - `flush_time_microseconds` from `system.asynchronous_insert_log`. Its interval - runs from the view-creation query's server start to a server timestamp after - snapshot completion. This is a server-observed envelope around those operations, - not row-level commit tracing or an exact microsecond activation timestamp. -- After writers stop, the reproduction flushes the remaining async queue before - counting missing IDs. A still-queued final batch is not evidence of data loss. -- UUID-mismatch `FlushError` entries can have `rows = 0`: rejection occurs before - payload rows are counted. Report error entries, bytes and query IDs, not an - inferred lost-row count from `sum(rows)`. - -The A/D observations match the mechanism above: A leaves IDs 1,2,3,5 in the new -name `t` and the dependent sink, and 1,2,3,4,5 in the old physical table. D1 leaves -20,000 early rows out of the new view, then copies 10 later rows. D2 copies the -three buffered rows through the newly created view; its 6,000-row cascade is -visible in QueryStart's table list. D4 preserves all 15,000 rows across the swap. -D5 rejects both queued batches with code 741, and its slow flush is visible as -`AsyncInsertFlush`. Cloud and the untested items above remain unverified. - -Final E measurement, after draining the stopped writers' async queue: - -| Trial | Preload rows | Writer rows: sync / async wait=1 / async wait=0 | Missing IDs | Duplicate IDs inside / outside server window | Final rows / distinct IDs | -|---|---:|---|---:|---|---| -| 1 | 6,000,000 | 218,000 / 8,600 / 25,600 | 0 | 100,000 / 0 | 6,352,200 / 6,252,200 | -| 2 | 6,000,000 | 178,000 / 7,000 / 21,000 | 0 | 59,000 / 0 | 6,265,000 / 6,206,000 | - -Both runs had zero duplicated preload rows, zero unattributed duplicate IDs, -zero writer errors and zero dependent-sink IDs with multiplicity other than one. -The final sorting key was `k, ts, id`. Reproduction command for these two results: - -```bash -env -u CH_MIGRATE_IT_URL uv run --locked --python 3.12 pytest -q -s \ - -m integration tests/integration/rebuild_spike_probe.py -k original_prototype -``` - -Result: **2 passed**. The combined A/D/E run before the final-drain measurement -correction passed **7 probes**. These measurements do not remove D5's separately -reproduced UUID-mismatch loss mode; the final two continuous-writer runs happened -not to lose a fire-and-forget batch. - -The local gate `uv run --locked --python pytest -q` passed **641 tests** -on each of Python 3.10, 3.11, 3.12, 3.13 and 3.14. The full owned-container suite -on Python 3.14 passed **78 tests**. No remote CI, release, or Cloud test is implied. diff --git a/docs/design/2026-10-02-waiting.md b/docs/design/2026-10-02-waiting.md index 54a4f95..a3e2da6 100644 --- a/docs/design/2026-10-02-waiting.md +++ b/docs/design/2026-10-02-waiting.md @@ -1,5 +1,13 @@ # Waiting for mutations and distributed DDL +**Superseded for 0.7 (Dan, 2026-10-27).** 0.7 doesn't ship the journal design below. It +waits by observation: after each write, `up` polls `system.mutations` until the project +database has none unfinished, and ClickHouse waits for `ON CLUSTER` DDL itself. A rerun +waits for unfinished work, then runs the migration again, relying on the idempotency gate. +Capability 5 in the [product definition](2026-10-02-ch-migrate-1.0.md) and the README's +"Migrations that wait" describe what ships. The rest of this note is the record of the +journal design and its experiments. + **DRY-1398 · 2026-10-02 · Recovery policy approved by Dan on 2026-10-04.** A write-ahead intent plus a marker carried into ClickHouse's mutation record can @@ -299,11 +307,11 @@ Rules: - Queue retention creates the same unknown-outcome boundary as Q5. A missing entry must not trigger blind DDL resubmission. -## Q9. Cloud: reasoned, then tested +## Q9. Cloud: reasoned, not tested -When this note was written (2026-10-02), DRY-1413 had no `service ready` comment. -No Cloud connection or credential lookup was made for the reasoned answers below. -They come from official documentation, not the local replicated harness: +DRY-1413 has no `service ready` comment. No Cloud connection or credential lookup +was made for this note. These conclusions come from official documentation, not +the local replicated harness: - The [system.mutations documentation](https://clickhouse.com/docs/reference/system-tables/mutations) says each Cloud node exposes all cluster mutations, so `clusterAllReplicas` is @@ -321,62 +329,6 @@ The 0.7 acceptance ticket must prove mutation visibility, marker preservation, retention behavior, session continuity and journal persistence on the authorized Cloud service. Local success cannot close these checks. -### Cloud results (2026-10-13) - -These come from the DRY Cloud test service: ClickHouse 26.6, two replicas, -Shared databases, harness user `chm_it`. - -- **Confirmed: every node lists every mutation.** `clusterAllReplicas` showed the - same `system.mutations` rows, ids and commands on both replicas. A single - node's table is enough there. -- **Confirmed: markers survive.** An owned `UPDATE ..., DELETE WHERE 0 AND - ('' = '')` shows as one `mutation_id` with one row per command, the - marker among them, on both replicas. The 0.7 waiting tests that run on Cloud - pass: the corpus waits, failed-mutation refusal, raw bind paths, foreign - failures, lost version-insert receipts and transport loss. A deliberately slow - mutation (`sleepEachRow(2)`) finished in `part_log` before the version row was - written (`test_cloud_up_advances_version_only_after_mutation_finishes`). -- **Confirmed, on one connection: a session reaches one replica.** In - `test_cloud_session_routing_keeps_set_for_every_statement`, 50 probes spread - over 98–103 s in three runs all read `max_threads = 3` from one earlier `SET`, and a single - replica served all of them. -- **Corrected: Cloud routes connections, not sessions.** A later probe sent one - `session_id` on twelve fresh TCP connections. They landed on both replicas, and the other - replica silently started a new session with default settings. ch-migrate sends - `session_check=1`, so such a request fails with `SESSION_NOT_FOUND` instead of running - without the `SET`. Two things could move a run to a new connection: an idle gap longer - than Cloud's 10-second `keep_alive_timeout`, and other in-process clients sharing the - driver's default connection pool. The second is how a rebuild heartbeat broke a Cloud run. - The migration session now has a single-connection pool of its own, and the rebuild keeps - it busy during its grace period. `X-ClickHouse-Replica-Tag` routing would pin it to a - replica, but that is an Enterprise beta and is not enabled on the test service. -- **Confirmed: SharedMergeTree conversion.** The version table is created as - `MergeTree` and reports `SharedMergeTree`; the journals share that engine - policy (`migration_state_engine`). No Keeper path is introduced. -- **Confirmed: crash-resume from the journal.** In - `test_cloud_killed_up_reattaches_to_its_running_mutation`, `up` was SIGKILLed while - a slow `UPDATE` (`sleepEachRow(3)`) was running. In 4 of 4 runs the rerun printed - "not reissuing SQL", resumed from the journal, left exactly one increment and one - owned mutation, and advanced the version. -- **Not tested on Cloud:** holding a mutation pending indefinitely. Cloud grants no - user `SYSTEM STOP MERGES`, so the timeout, held-resume, foreign-predecessor and - progress tests skip there and remain self-hosted evidence. Retention of mutation - evidence was not exercised. -- **New: metadata ALTERs leave a `system.mutations` row (DRY-1446).** On - SharedMergeTree, every corpus ALTER classified `metadata` (among them `ADD - COLUMN`, `MODIFY COLUMN ... DEFAULT`, `ADD INDEX`, `ADD PROJECTION`, `MODIFY TTL` - without materialization, `REMOVE TTL`) creates a row with no block numbers - and `parts_to_do = 0`. On the initiating replica it can read `is_done = 0` for - a few milliseconds. Ids are shared with real mutations (`0000000000` metadata, - `0000000001` the next `UPDATE`). Self-hosted MergeTree creates no such row. - **Decision: keep the classifier; these rows rewrite no parts.** Waiting binds - ownership only to rows that carry the statement's token, so these rows are - never adopted. If one is still `is_done = 0` when a later statement prepares, - `prepare` waits for it as a foreign predecessor, which is a millisecond pause - and the same conservative rule as self-hosted. `plan` checks unfinished - mutations only on `alembic_version`, which ch-migrate never ALTERs. The tests - count only part-rewriting mutations and say so for Shared engines. - ## Which statements create mutation records The corpus experiment runs each statement on a fresh seeded MergeTree table and diff --git a/docs/design/2026-10-13-rebuild-on-cloud.md b/docs/design/2026-10-13-rebuild-on-cloud.md deleted file mode 100644 index 0fe1b6b..0000000 --- a/docs/design/2026-10-13-rebuild-on-cloud.md +++ /dev/null @@ -1,169 +0,0 @@ -# Rebuilding tables on ClickHouse Cloud - -**Date:** 2026-10-13 -**Status:** Cloud support chosen and the protocol below approved (Dan, 2026-10-13): all four points, with plain helpers on self-hosted too and the required `writers_on_this_service_only` declaration plus grace period. Implemented under DRY-1445; see [Implementation and Cloud results](#implementation-and-cloud-results). - -## Decision - -ClickHouse Cloud supplies a consistent Shared catalog and immutable SharedMergeTree parts. It does **not** supply the three application-level fences used by the current `rebuild_table`: caller-chosen UUIDs, a frozen snapshot-helper part set, or an async-insert queue barrier at `EXCHANGE`. **Dan decided on 2026-10-13 that 1.0 fully supports Cloud, `rebuild_table` included**, as the [accepted 1.0 definition](2026-10-02-ch-migrate-1.0.md#8-oprebuild_table) already promises. The protocol below replaces each fence with one Cloud allows, keeps every fail-closed recovery rule, and lists what Dan is asked to approve. - -**Before implementation:** `plan` reported and `up` refused a rebuild on a Shared -database with the preflight finding `cloud_unsupported`, before any lock, helper or -swap existed. That refusal is gone; the undeclared-writers refusal replaces it. - -## What the current engine protects - -1. **Explicit UUIDs.** `_Lock._create` embeds the lock UUID, owner comment and isolated Keeper path; `_owned` refuses another owner's lock across hosts. `_initialize` records the old source UUID and three planned helper UUIDs *before* DDL; `helper_ddl` and `dual_ddl` embed UUIDs in CREATE. `_Context.ddl` persists intent before sending, refuses an unowned pre-existing result, and accepts a checkpointed result only on all-host UUID convergence. `_dual`, `_resolve_exchange`, `_cleanup`, `_result` and `_verify_complete` identify the owned view/helpers, prove the old source moved to the rollback name and the replacement took the live name, rather than trusting names alone. `_async_report` filters `FlushError` exceptions containing the **old UUID**. See `ch_migrate/rebuild_engine.py:_Lock._create,_Lock._owned,_initialize,_Context.ddl,_Context.uuid_map,_Context.converged,_dual,_exchange_evidence,_resolve_exchange,_cleanup,_async_report,_result,_verify_complete` and `ch_migrate/rebuild_ddl.py:helper_ddl,dual_ddl`. A `COMMENT` identifies the lock owner, but the UUID is also checked; a name or comment alone is not its current proof. -2. **`SYSTEM STOP MERGES` on `snap` and `stage`, not the live source.** `_helpers` stops both, `_snapshot` re-stops `snap` on restart, and `_copy` re-stops `stage`. `_snapshot` freezes per-partition row counts after attaching each source partition; `_copy_partition` uses those counts to distinguish an incomplete stage, a complete staged partition, and a moved partition without replaying an ambiguous MOVE. `_verify_stage` checks stage partition IDs and those counts on every replica. A `Replacing`/`Collapsing`/`Summing`/`Aggregating` merge on **snap** can change the physical row count or values between count capture and SELECT, making the stage-count equality impossible or, worse, making a count-only test insufficient to prove content. A merge on **stage** can change the count between verification and MOVE and invalidate the `{0, expected}` recovery states; even count-preserving merges change part names, but this code does **not** journal part names. Source merges are allowed; immutable attached source parts remain readable. See `ch_migrate/rebuild_engine.py:_helpers,_stop_merges,_snapshot,_copy,_copy_partition,_verify_stage` and [parts and merge model](https://clickhouse.com/docs/parts). -3. **`SYSTEM FLUSH ASYNC INSERT QUEUE`.** `_swap` flushes the old live name immediately before `EXCHANGE`, reducing but not closing the window for queued pre-swap writes. `_async_report` later flushes the **rollback name/old UUID**, drains pre-barrier processes, flushes logs, and bounds the old-UUID `FlushError` query by `swap_intent`/`report_end`. Without that second barrier, `async_flush_errors` could undercount errors not yet emitted, so `_result` cannot honestly report a final count, bytes or query IDs. Neither flush is atomic with concurrent enqueue. See `ch_migrate/rebuild_engine.py:_swap,_flush_async,_async_report,_drain_before,_result` and [D5 in the rebuild spike](2026-10-02-rebuild-spike.md#d-insert-races). - -## Cloud guarantees and limits - -- **Catalog identity: partial.** [Shared Catalog documentation](https://clickhouse.com/docs/cloud/reference/shared-catalog) describes centralized versioned database/table metadata and replica convergence. The [architecture account](https://clickhouse.com/blog/clickhouse-cloud-stateless-compute#stage-3-decoupling-metadata-with-a-shared-catalog) describes `/uuids` (object metadata), `/names` (name-to-UUID) and `/replicas` (applied version), with linearizable Keeper multi-writes for conflicting DDL. The [Atomic engine reference](https://clickhouse.com/docs/engines/database-engines/atomic) describes persistent UUIDs independent of names. Cloud assigns an object UUID and preserves it across `EXCHANGE` (probe below). It does **not** accept caller-assigned UUIDs in Shared databases: explicit table/view `UUID` returns Code 36, and `database_replicated_allow_explicit_uuid` is read-only; explicit replica name in SharedMergeTree arguments returns Code 80 ([Cloud spike](2026-10-02-rebuild-spike.md), DRY Cloud test service results, 2026-10-13). Catalog serializability is not an application lock or a durable, caller-owned CREATE receipt. `log_comment`/`query_id` in [query_log](https://clickhouse.com/docs/operations/system-tables/query_log) attribute a request, not a surviving object; log retention and per-node logging prevent treating absence as proof that CREATE did not run. A unique `COMMENT` plus issuer token in persisted intent can distinguish an owned result *after* catalog visibility, but a vanished object or an unacknowledged CREATE with no observable result remains unknown. -- **Parts and transfers: partial.** [SharedMergeTree](https://clickhouse.com/docs/cloud/reference/shared-merge-tree) shares storage and Keeper part metadata; replica metadata fetch is asynchronous, and `SYSTEM SYNC REPLICA … LIGHTWEIGHT` can fetch it. [MergeTree parts](https://clickhouse.com/docs/parts) are immutable; a merge publishes a new part rather than modifying the attached input. [ALTER PARTITION](https://clickhouse.com/docs/reference/statements/alter/partition#attach-partition-from) promises a copy with the source intact, and [MOVE PARTITION](https://clickhouse.com/docs/reference/statements/alter/partition#move-partition-to-table) removes from source. The DRY Cloud test service accepts both between same-schema SharedMergeTree tables and converges on two replicas (the prior A/B spike). Neither documentation nor that observation proves that every SharedMergeTree transfer is metadata-only or that cross-table MOVE is a single atomic transaction through a server crash. Do **not** sell the local hardlink timing from [the self-hosted spike](2026-10-02-rebuild-spike.md#b-attach-partition-from-and-move-partition) as a Cloud zero-copy guarantee. Immutable parts protect a snapshot already attached; they do not freeze `ReplacingMergeTree` semantics or a helper's active-part set. `SYSTEM SYNC REPLICA` synchronizes visibility, not merges or async-insert buffers. -- **Async DDL: no swap barrier.** [Async insert documentation](https://clickhouse.com/docs/concepts/features/operations/insert/asyncinserts) says buffers are per node and shape; flush rechecks destination catalog/authorization. `wait_for_async_insert=1` acknowledges after flush and returns errors, while `wait_for_async_insert=0` acknowledges before flush and has no delivery guarantee or dead-letter queue. It documents a server-wide `SYSTEM FLUSH ASYNC INSERT QUEUE` for maintenance, not an automatic queue flush on `EXCHANGE`/`DROP`. On the DRY Cloud test service, `SYSTEM FLUSH ASYNC INSERT QUEUE` (including the table-scoped form used by the engine) requires unavailable `SYSTEM FLUSH ASYNC INSERT QUEUE ON *.*` privilege (Code 497); Cloud does not supply that fence to this operator. A prior queued-swap probe observed Code 741 `TABLE_UUID_MISMATCH` and lost fire-and-forget rows. `insert_deduplication_token` is a [bounded retry/dedup aid](https://clickhouse.com/docs/concepts/features/operations/insert/deduplicating-inserts-on-retries), not a queue flush or an app-level migration lock. [asynchronous_inserts](https://clickhouse.com/docs/operations/system-tables/asynchronous_inserts), [asynchronous_insert_log](https://clickhouse.com/docs/operations/system-tables/asynchronous_insert_log), `query_log` and `system.processes` are per node; inspect all replicas, with `skip_unavailable_shards=0` when completeness is required. Their existence does not prove absence of writes from another compute service. - -## Synthetic Cloud observations - -The DRY Cloud test service ran ClickHouse 26.6.1.2326 with two replicas and a Shared database. The following are **observations**, not general Cloud contracts; all database names are replaced by `it_design_db` and UUIDs by `U_old`/`U_new`. Client credentials are not recorded. - -| SQL / sequence | Observed result | -|---|---| -| `SELECT name,value FROM system.settings WHERE name IN ('async_insert','wait_for_async_insert','async_insert_busy_timeout_ms','async_insert_use_adaptive_busy_timeout')` | For `chm_it`: `1`, `1`, `1000`, `1`, respectively. This is the service/user effective default, not a guarantee for other writer profiles. | -| `CREATE TABLE it_design_db.t (id UInt64,p UInt8) ENGINE=MergeTree PARTITION BY p ORDER BY id COMMENT 'design-owner-token'` with setting `log_comment='design-ddl-token'`; query `clusterAllReplicas('default',system.tables)` | Both replicas returned the same server-assigned `U_old` and comment. After `EXCHANGE TABLES it_design_db.t AND it_design_db.next`, both showed `U_old` and its comment under `next`, and `U_new` under `t`. | -| `SYSTEM FLUSH LOGS`; query `clusterAllReplicas('default',system.query_log)` for `log_comment='design-ddl-token'` | `QueryStart`/`QueryFinish` entries with the CREATE text and token were present (two node-local query IDs observed). This proves positive request attribution in this run, not an always-available durable receipt. | -| `SYSTEM SYNC REPLICA it_design_db.t LIGHTWEIGHT` | Succeeded for `chm_it`; SharedMergeTree documentation permits it for read-after-remote-write metadata visibility. | -| Three inserts into partitions 1 and 2; `ALTER TABLE it_design_db.snap ATTACH PARTITION ID '1' FROM it_design_db.t`; `OPTIMIZE TABLE it_design_db.t PARTITION ID '2' FINAL`; `ALTER TABLE it_design_db.snap ATTACH PARTITION ID '2' FROM it_design_db.t` | Both ATTACH steps succeeded and both replicas read `snap` as eight rows, sum(id)=241 (six source rows plus two independently inserted snap rows). The `OPTIMIZE` command succeeded, but this probe did not prove its source merge completed before the second ATTACH. Its immediate `system.parts` aggregation failed; the separate helper-merge probe below does prove a changed part set. | -| `CREATE TABLE it_design_db.source/snap (…) ENGINE=ReplacingMergeTree(v) PARTITION BY p ORDER BY id`; two inserts with `(id,v,p)=(1,1,1),(1,2,1)`; `ALTER TABLE snap ATTACH PARTITION ID '1' FROM source`; `OPTIMIZE TABLE snap PARTITION ID '1' FINAL`; `SYSTEM SYNC REPLICA snap LIGHTWEIGHT` | Before the optimize, both tables showed two active parts and two physical rows in partition `1`. After convergence, the source retained parts `1_0_0_1`,`1_1_1_1` and count 2, while `snap` had part `1_0_1_2` and count 1 (sum(v)=2); replica-wide `system.parts` reported the same merged part on both replicas. The helper merge changed row count, value set and part names **without** changing the source. | -| `CREATE TABLE … a0/b0` and `a1/b1` (same schema), `INSERT INTO a0/a1 VALUES (…)` with `async_insert=1, wait_for_async_insert=0/1, async_insert_use_adaptive_busy_timeout=0, async_insert_busy_timeout_ms=15000`; observe `clusterAllReplicas('default',system.asynchronous_inserts)` nonempty, then `EXCHANGE TABLES aN AND bN` | For `wait=0`, queue held five bytes on one node and client had already received `ACK`; both tables subsequently had zero rows. For `wait=1`, queue held five bytes, client had **not** received ACK at swap, then received Code 741; both tables had zero rows. The logged wait=1 entry was `FlushError` with `TABLE_UUID_MISMATCH`. A fire-and-forget `FlushError` log entry was not observed in this sample: never equate missing logs with absence of loss. | -| Earlier A/B/D4/D5 Cloud spike: `EXCHANGE`, `ATTACH PARTITION FROM`, `MOVE PARTITION TO TABLE`, in-flight insert across swap then `DROP TABLE dual SYNC` | Same name-following MV, accepted transfers and two-replica visibility as self-hosted; in-flight insert landed fully and DROP waited about 13 seconds. The end-to-end rebuild stopped at the denied queue flush. These observations do not prove arbitrary concurrent-load or crash recovery. | -| Prior Cloud spike: `CREATE TABLE … UUID '…'` / `CREATE MATERIALIZED VIEW … UUID '…'`; `SET database_replicated_allow_explicit_uuid=1` (or `2`); `CREATE TABLE … ENGINE=ReplicatedMergeTree('path','replica')`; `SYSTEM STOP MERGES it_design_db.snap`; `SYSTEM FLUSH ASYNC INSERT QUEUE it_design_db.t` | Explicit UUID rejected (Code 36) in Shared; setting is read-only; explicit replica name rejected (Code 80); STOP MERGES access denied and not grantable; table-scoped queue flush access denied (Code 497, requires grant `ON *.*`). These are results on this service, not proof about future Cloud releases. | - -### Protocol probes (2026-10-13, same service) - -| Probe | Result | -|---|---| -| Replacing, Collapsing, Aggregating and Summing source (with TTL) → `ATTACH PARTITION ID '1' FROM` into a **plain** `MergeTree` snap with the same columns, ORDER BY and PARTITION BY and no TTL; `OPTIMIZE ... FINAL` on snap; `INSERT ... SELECT` into a plain stage; `OPTIMIZE ... FINAL`; `MOVE PARTITION` stage → a target with the source's engine | Every step accepted for all four families. The plain helpers kept all 3 physical rows through `OPTIMIZE FINAL` on both replicas, and the MOVE delivered 3 rows to the family target and emptied the stage. | -| `CREATE TABLE ... COMMENT ''`, then `EXCHANGE` | Both replicas show one server UUID with the token. After `EXCHANGE`, UUID and comment travel together under the other name. `ALTER TABLE ... MODIFY COMMENT` is allowed, so an edited comment must count as lost ownership. | -| `CREATE MATERIALIZED VIEW v TO dst COMMENT '' AS SELECT ...` | Accepted; the token shows in `system.tables.comment` on both replicas and the view delivers rows. `... AS SELECT ... FROM src COMMENT '...'` fails (62) because `COMMENT` parses as a table alias, so put `COMMENT` before `AS`. | -| `SYSTEM SYNC REPLICA ON CLUSTER 'default' t LIGHTWEIGHT`, `KILL QUERY ON CLUSTER 'default' ... SYNC`, `SYSTEM FLUSH LOGS ON CLUSTER 'default'`; `system.clusters` | All accepted for `chm_it`. Cluster `default` has 1 shard and 2 replicas. | -| Six concurrent async inserts (three `wait=0`, three `wait=1`, busy timeout 30 s) queued across both replicas, then `EXCHANGE` with **no** queue flush; three trials | The old table's queue entries (`system.asynchronous_inserts`, cluster-wide) were gone 0.15–0.23 s after `EXCHANGE`, long before the busy timeout. All `wait=0` inserts were acknowledged and lost; all `wait=1` inserts got 741. After `SYSTEM FLUSH LOGS ON CLUSTER`, `asynchronous_insert_log` held old-UUID `FlushError` rows for 6, 5 and 6 of the 6 queued inserts. | -| Self-hosted 26.3: `SYSTEM STOP MERGES` on a ReplacingMergeTree, two inserts, restart the server, `OPTIMIZE FINAL` | Before restart the merge was refused (236) and 2 physical rows remained. **After restart the merge ran and 1 row remained: `STOP MERGES` does not survive a restart.** | - -## Proposed Cloud protocol (for approval) - -### 1. Ownership: owner-token comment, then read back the UUID - -On a Shared database, `_initialize` no longer generates helper UUIDs. It journals a token for each role (`chm-rebuild:::`) before any CREATE. The lock, `new`, `snap`, `stage` and the dual view are created **without** `UUID` and **with** `COMMENT ''`, using plain engine names that Cloud converts to Shared, with no Keeper path. For the view, `COMMENT` goes before `AS`. - -An object is ours only when every replica (`clusterAllReplicas('default', system.tables)`) shows the name with exactly our token and one common UUID. That UUID is then journaled ("bound"). From that point every existing UUID check runs unchanged against the bound value: exchange evidence, rollback identity, cleanup, old-UUID async attribution and the result. - -| Resume finds | Action | Rule | -|---|---|---| -| Intent journaled, name absent, CREATE not acknowledged | `UnknownOutcome`; operator reconciles | Same as today for unacknowledged local DDL | -| Name with our token on every replica, UUID not yet bound | Bind and continue | The catalog proves ownership; covers a crash between acknowledgement and bind | -| Name with another comment, or replicas disagree on UUID | `UnknownOutcome`; never adopt | Same as the unowned pre-existing rule | -| Bound object whose comment changed later | Ownership lost; fail closed | New case; there is no takeover path | - -The lock keeps its heartbeat and the 2026-10-04 rule: expiry never authorizes takeover; the operator confirms and releases. Self-hosted keeps chosen UUIDs, because its helper Keeper paths embed them. - -### 2. Snapshot without `STOP MERGES`: plain, TTL-free helpers - -`snap` and `stage` become plain `MergeTree`, in the same replication family as the source (Shared on Cloud; `ReplicatedMergeTree` with a distinct Keeper path on replicated self-hosted). They keep the same columns, ORDER BY, PARTITION BY, PRIMARY KEY, indexes and projections, but drop engine parameters and every table and column TTL. Plain merges keep every physical row, and with no TTL nothing expires. The per-partition counts taken at the snapshot therefore stay exact however often the helpers merge, and the `{0, expected}` MOVE recovery states stay valid. `new` keeps the target engine. Preflight compares source → snap and stage → target for transfer compatibility (columns, ORDER BY, partition key) instead of requiring equal engines for the helpers. - -**Proposed for self-hosted as well, deleting `STOP MERGES`.** The restart probe shows `STOP MERGES` silently lapses when a node restarts. After such a restart, a Replacing, Collapsing, Summing or Aggregating snapshot can merge away rows, so `_verify_stage` never sees the count it expects. [INFERENCE] The run would then fail closed and need an operator, rather than lose rows. One mechanism that holds by construction is simpler than two. Cost: DRY-1410's continuous-writer acceptance (5 variants × 5 runs) must be re-run on the new helpers. - -### 3. Swap without a forced queue flush - -On Cloud the pre-`EXCHANGE` flush is skipped. `EXCHANGE` itself drained the old queue within about 0.2 s in every trial. The guarantee is unchanged: synchronous and `wait=1` rows are never lost (`wait=1` writers get 741 and retry), and `wait=0` writers still need `allow_unacknowledged_async_loss=True`. - -After the swap, the flush of the rollback table is replaced by a cluster-wide wait: no `system.asynchronous_inserts` entry for the table with `first_update` at or before the `EXCHANGE` acknowledgement may remain on any replica. Then come the existing drain of in-flight flushes, `SYSTEM FLUSH LOGS ON CLUSTER 'default'`, and the old-UUID `FlushError` query, repeated until two consecutive reads agree. On Cloud the report says `at least N` and names the replicas it observed, because one trial logged 5 of 6. Self-hosted keeps the flush. - -### 4. Topology and other writers - -For a Shared database the engine inspects through cluster `default`: hosts, catalog, processes, counts, logs, `KILL QUERY` and `SYSTEM SYNC REPLICA ... LIGHTWEIGHT`, all accepted in the probes. DDL never uses `ON CLUSTER`. Source and `new` are synchronized on Shared as they are on Replicated today. - -**Gap: writers on other compute services.** Under Cloud compute-compute separation, another service's replicas, queues and running inserts are invisible to this one. An insert from such a service that started before the dual view reached it would skip the view. If it commits after the snapshot, an acknowledged row would be missing from the new table. That case breaks the guarantee, and ch-migrate cannot detect it from inside the service. Proposal: on Cloud the migration must declare `writers_on_this_service_only=True`. Like the async-loss opt-in, there is no command-line bypass. It comes with a configurable minimum grace period after the dual view is visible, and `plan` states that the condition cannot be checked. - -### Acceptance for this protocol - -- **Unit:** - - every row of the ownership table; - - plain, TTL-free helper DDL for each engine family; - - the queue-drain and report logic. -- **Docker, single node and cluster:** - - all rebuild suites; - - DRY-1410's five continuous-writer variants, five consecutive runs; - - a new node-restart-during-copy test. -- **Cloud:** - - scaffolded rebuild round trip; - - the rebuild command and dependent suites; - - continuous-writer variants `sync`, `async1` and `async0` against the service; - - SIGKILL mid-copy, then operator release and resume; - - rebuild spike E re-run to completion and recorded in rebuild-spike section C. -- The `cloud_unsupported` refusal is removed only when all of the above pass. - -### What approval covers - -1. Token-comment ownership with read-back UUIDs on Shared databases (self-hosted unchanged). -2. Plain, TTL-free `snap`/`stage` helpers everywhere, deleting `STOP MERGES`, plus the DRY-1410 re-run. -3. A Cloud swap without forced flushes, with the async report as an observed lower bound. -4. The required `writers_on_this_service_only=True` declaration plus grace period on Cloud, or another way to handle other-service writers. - -## Implementation and Cloud results - -Implemented on `dy/ch-migrate-1.0` (DRY-1445): - -| Commit | Change | -|---|---| -| `f704685` | Plain, TTL-free helpers; `STOP MERGES` deleted | -| `e320f57` | The migration session gets its own single-connection pool | -| `7409914` | Preflight filters and groups the writer log in the server | -| `1dbe50f` | The Cloud protocol: tokens, no forced flush, declaration and grace | -| `41b69ce` | Test attribution fixes found during acceptance | -| `bb3772e` | Stranded-row check (below) | - -Two problems surfaced only on Cloud and were fixed before acceptance: - -- **The session pool.** Rebuild heartbeats shared clickhouse-connect's default pool with the migration session, so a second connection opened under it. Cloud routes each connection, not each session id, and the next request answered `SESSION_NOT_FOUND`. -- **The writer log.** Preflight capped all inserts on the service at 10,001 rows before filtering to the table, and refused on a busy service. - -### Added after approval: rows stranded at the swap - -An insert that resolves the table just before `EXCHANGE` and builds its view chain just after it writes only to the old storage, and the copying view never sees it. This is not Cloud-specific. A probe (`spikes/2026-10-13-exchange-race/probe.py`) ran eight writers against one `EXCHANGE` per trial: - -| Server | Writers | Trials with loss | Rows lost and found only in the old storage | -|---|---|---|---| -| 26.3 | sync | 0 / 160 | 0 of about 60,000 | -| 26.3 | async, `wait=1` | 3 / 160 | 24 | -| Cloud 26.6 | async, `wait=1` | 1 / 40 | 1 | - -The same race is the likely cause of a CI failure in which the `async0` report missed one lost batch. That batch's flush logged `Ok`, not `FlushError`, and its rows were not in the live table; the rollback table was not inspected. In all 240 trials where the probe checked it (26.3 sync and `wait=1`, Cloud `wait=1`), `system.part_log` identified the stranded rows exactly: `NewPart` in the old UUID after the swap, from a query with no `NewPart` in the new UUID. - -Dan chose detect-and-stop (2026-10-13): - -- After the async report, `_stranded` runs that check. -- Any stranded rows stop the run with their count and query IDs, and the run keeps its lock. -- The operator copies the rows and releases the lock; the next run records the stop as acknowledged and finishes. -- Preflight refuses a host without `system.part_log`. - -The guarantee is now: no acknowledged row is *silently* lost. - -### Acceptance (all at `bb3772e` unless noted) - -| Check | Result | -|---|---| -| Unit, Python 3.10–3.14 | 727 passed on each | -| Docker `integration`, cluster included | 191 passed, 4 skipped (Cloud-only) | -| DRY-1410 continuous writers on Docker, 5 variants × 5 runs | 25 / 25 | -| Docker node-restart-during-copy (new) | passed | -| Cloud `(integration and not cluster) or cloud` | 152 passed, 0 failed, 19 skipped | -| Cloud continuous writers `sync`, `kill`, `async1`, `async0` | 4 / 4 (at `1dbe50f`; included in the full run) | -| Cloud stranded-row stop and operator resume | passed | -| Rebuild spike E without the flush | complete; rebuild-spike section C | - -The 19 Cloud skips are 17 tests that need `SYSTEM STOP MERGES` to hold a mutation, the Docker-only helper-merge test, and a test that changes `log_query_settings` in a profile. - -Remote CI on PR #14 at `bb3772e`: unit jobs passed. The cluster job failed once with `TABLE_IS_READONLY` (242) on a just-created replicated journal in `test_continuous_insert_replicated`, then passed on re-run. That job's `test_cluster_harness` restarts node 2 first; the readonly journal is on node 1, whose embedded Keeper never stopped, so the cause is open. - -An earlier `async0` failure mode, duplicates logged 6 ms after the window closed, was a test attribution error. It also occurred at `1d512b7`, in 2 of 20 runs; a third run there failed with the stranded-flush loss above. The test now treats an async insert as in flight from queueing to the end of its flush (`41b69ce`): 25 of 25 runs passed. diff --git a/docs/design/spikes/2026-10-02-rebuild/a_exchange_mv.sql b/docs/design/spikes/2026-10-02-rebuild/a_exchange_mv.sql index ad6fa40..be7c69f 100644 --- a/docs/design/spikes/2026-10-02-rebuild/a_exchange_mv.sql +++ b/docs/design/spikes/2026-10-02-rebuild/a_exchange_mv.sql @@ -29,5 +29,5 @@ INSERT INTO a.t VALUES (3, 'post-direct-into-name-t'); INSERT INTO a.t_new VALUES (4, 'post-direct-into-name-t_new'); INSERT INTO a.feeder VALUES (5, 'post-feeder'); -SELECT * FROM (SELECT 'name t' AS tbl, id, src FROM a.t UNION ALL SELECT 'name t_new' AS tbl, id, src FROM a.t_new) ORDER BY tbl, id; +SELECT 'name t' AS tbl, id, src FROM a.t UNION ALL SELECT 'name t_new', id, src FROM a.t_new ORDER BY tbl, id; SELECT 'sink_src' AS tbl, id, src, via FROM a.sink_src ORDER BY id; diff --git a/docs/design/spikes/2026-10-02-rebuild/cloud/a.out b/docs/design/spikes/2026-10-02-rebuild/cloud/a.out deleted file mode 100644 index 456f824..0000000 --- a/docs/design/spikes/2026-10-02-rebuild/cloud/a.out +++ /dev/null @@ -1,137 +0,0 @@ -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -DROP DATABASE IF EXISTS it_spike_a SYNC; -CREATE DATABASE it_spike_a ENGINE = Shared; -CREATE TABLE it_spike_a.t (id UInt64, src String) ENGINE = MergeTree ORDER BY id; -CREATE TABLE it_spike_a.t_new (id UInt64, src String) ENGINE = MergeTree ORDER BY (src, id); -CREATE TABLE it_spike_a.feeder (id UInt64, src String) ENGINE = MergeTree ORDER BY id; -CREATE TABLE it_spike_a.sink_src (id UInt64, src String, via String) ENGINE = MergeTree ORDER BY id; --- MV whose SOURCE is t -CREATE MATERIALIZED VIEW it_spike_a.mv_src TO it_spike_a.sink_src AS SELECT id, src, 'mv_src' AS via FROM it_spike_a.t; --- MV whose TARGET is t -CREATE MATERIALIZED VIEW it_spike_a.mv_tgt TO it_spike_a.t AS SELECT id, src FROM it_spike_a.feeder; --- the dual-write MV: source t, target t_new -CREATE MATERIALIZED VIEW it_spike_a.t_dual TO it_spike_a.t_new AS SELECT id, src FROM it_spike_a.t; -SELECT name, uuid FROM system.tables WHERE database = 'it_spike_a' AND name IN ('t','t_new') ORDER BY name; - ┌─name──┬─uuid─────────────────────────────────┐ -1. │ t │ 8c8cc67e-fde0-4cb9-baa2-006f0d4d52a1 │ -2. │ t_new │ d85ab0f6-f249-48c2-880f-a70aad286a71 │ - └───────┴──────────────────────────────────────┘ -SELECT name, dependencies_table, loading_dependencies_table, loading_dependent_table FROM system.tables WHERE database='it_spike_a' ORDER BY name; - ┌─name─────┬─dependencies_table──┬─loading_dependencies_table─┬─loading_dependent_table─┐ -1. │ feeder │ ['mv_tgt'] │ [] │ [] │ -2. │ mv_src │ [] │ [] │ [] │ -3. │ mv_tgt │ [] │ [] │ [] │ -4. │ sink_src │ [] │ [] │ [] │ -5. │ t │ ['t_dual','mv_src'] │ [] │ [] │ -6. │ t_dual │ [] │ [] │ [] │ -7. │ t_new │ [] │ [] │ [] │ - └──────────┴─────────────────────┴────────────────────────────┴─────────────────────────┘ -INSERT INTO it_spike_a.t VALUES (1, 'pre-direct'); -INSERT INTO it_spike_a.feeder VALUES (2, 'pre-feeder'); -EXCHANGE TABLES it_spike_a.t AND it_spike_a.t_new; -SELECT name, uuid FROM system.tables WHERE database = 'it_spike_a' AND name IN ('t','t_new') ORDER BY name; - ┌─name──┬─uuid─────────────────────────────────┐ -1. │ t │ d85ab0f6-f249-48c2-880f-a70aad286a71 │ -2. │ t_new │ 8c8cc67e-fde0-4cb9-baa2-006f0d4d52a1 │ - └───────┴──────────────────────────────────────┘ -SELECT name, dependencies_table FROM system.tables WHERE database='it_spike_a' ORDER BY name; - ┌─name─────┬─dependencies_table──┐ -1. │ feeder │ ['mv_tgt'] │ -2. │ mv_src │ [] │ -3. │ mv_tgt │ [] │ -4. │ sink_src │ [] │ -5. │ t │ ['mv_src','t_dual'] │ -6. │ t_dual │ [] │ -7. │ t_new │ [] │ - └──────────┴─────────────────────┘ -SHOW CREATE TABLE it_spike_a.mv_src FORMAT TSVRaw; -CREATE MATERIALIZED VIEW it_spike_a.mv_src TO it_spike_a.sink_src -( - `id` UInt64, - `src` String, - `via` String -) -AS SELECT - id, - src, - 'mv_src' AS via -FROM it_spike_a.t -SHOW CREATE TABLE it_spike_a.mv_tgt FORMAT TSVRaw; -CREATE MATERIALIZED VIEW it_spike_a.mv_tgt TO it_spike_a.t -( - `id` UInt64, - `src` String -) -AS SELECT - id, - src -FROM it_spike_a.feeder -SHOW CREATE TABLE it_spike_a.t_dual FORMAT TSVRaw; -CREATE MATERIALIZED VIEW it_spike_a.t_dual TO it_spike_a.t_new -( - `id` UInt64, - `src` String -) -AS SELECT - id, - src -FROM it_spike_a.t -INSERT INTO it_spike_a.t VALUES (3, 'post-direct-into-name-t'); -INSERT INTO it_spike_a.t_new VALUES (4, 'post-direct-into-name-t_new'); -INSERT INTO it_spike_a.feeder VALUES (5, 'post-feeder'); -SELECT * FROM (SELECT 'name t' AS tbl, id, src FROM it_spike_a.t UNION ALL SELECT 'name t_new' AS tbl, id, src FROM it_spike_a.t_new) ORDER BY tbl, id; - ┌─tbl────────┬─id─┬─src─────────────────────────┐ -1. │ name t │ 1 │ pre-direct │ -2. │ name t │ 2 │ pre-feeder │ -3. │ name t │ 3 │ post-direct-into-name-t │ -4. │ name t │ 5 │ post-feeder │ -5. │ name t_new │ 1 │ pre-direct │ -6. │ name t_new │ 2 │ pre-feeder │ -7. │ name t_new │ 3 │ post-direct-into-name-t │ -8. │ name t_new │ 4 │ post-direct-into-name-t_new │ -9. │ name t_new │ 5 │ post-feeder │ - └────────────┴────┴─────────────────────────────┘ -SELECT 'sink_src' AS tbl, id, src, via FROM it_spike_a.sink_src ORDER BY id; - ┌─tbl──────┬─id─┬─src─────────────────────┬─via────┐ -1. │ sink_src │ 1 │ pre-direct │ mv_src │ -2. │ sink_src │ 2 │ pre-feeder │ mv_src │ -3. │ sink_src │ 3 │ post-direct-into-name-t │ mv_src │ -4. │ sink_src │ 5 │ post-feeder │ mv_src │ - └──────────┴────┴─────────────────────────┴────────┘ -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -SELECT * FROM (SELECT 'name t (uuid c202..)' AS tbl, id, src FROM it_spike_a.t UNION ALL SELECT 'name t_new (uuid 74a5..)' AS tbl, id, src FROM it_spike_a.t_new) ORDER BY tbl, id; - ┌─tbl──────────────────────┬─id─┬─src─────────────────────────┐ -1. │ name t (uuid c202..) │ 1 │ pre-direct │ -2. │ name t (uuid c202..) │ 2 │ pre-feeder │ -3. │ name t (uuid c202..) │ 3 │ post-direct-into-name-t │ -4. │ name t (uuid c202..) │ 5 │ post-feeder │ -5. │ name t_new (uuid 74a5..) │ 1 │ pre-direct │ -6. │ name t_new (uuid 74a5..) │ 2 │ pre-feeder │ -7. │ name t_new (uuid 74a5..) │ 3 │ post-direct-into-name-t │ -8. │ name t_new (uuid 74a5..) │ 4 │ post-direct-into-name-t_new │ -9. │ name t_new (uuid 74a5..) │ 5 │ post-feeder │ - └──────────────────────────┴────┴─────────────────────────────┘ diff --git a/docs/design/spikes/2026-10-02-rebuild/cloud/b1.out b/docs/design/spikes/2026-10-02-rebuild/cloud/b1.out deleted file mode 100644 index b7ecb88..0000000 --- a/docs/design/spikes/2026-10-02-rebuild/cloud/b1.out +++ /dev/null @@ -1,90 +0,0 @@ -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -DROP DATABASE IF EXISTS it_spike_b SYNC; -CREATE DATABASE it_spike_b; -CREATE TABLE it_spike_b.t (id UInt64, ts DateTime, payload String) ENGINE = MergeTree PARTITION BY toYYYYMM(ts) ORDER BY id; -INSERT INTO it_spike_b.t SELECT number, toDateTime('2026-01-01 00:00:00') + intDiv(number, 50000) * 3600 * 24, repeat('x', 40) FROM numbers(5000000) SETTINGS max_partitions_per_insert_block = 1000; -Received exception from server (version 26.6.1): -Code: 452. DB::Exception: Received from :9440. DB::Exception: Setting max_partitions_per_insert_block should not be changed. (SETTING_CONSTRAINT_VIOLATION) -(query: INSERT INTO it_spike_b.t SELECT number, toDateTime('2026-01-01 00:00:00') + intDiv(number, 50000) * 3600 * 24, repeat('x', 40) FROM numbers(5000000) SETTINGS max_partitions_per_insert_block = 1000;) -OPTIMIZE TABLE it_spike_b.t FINAL; -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -SELECT partition, count() AS parts, sum(rows) AS rows, formatReadableSize(sum(bytes_on_disk)) AS size FROM system.parts WHERE database='it_spike_b' AND table='t' AND active GROUP BY partition ORDER BY partition; --- t_snap: same structure as t -CREATE TABLE it_spike_b.t_snap AS it_spike_b.t; -SET send_logs_level = 'none'; -ALTER TABLE it_spike_b.t_snap ATTACH PARTITION 202601 FROM it_spike_b.t; -ALTER TABLE it_spike_b.t_snap ATTACH PARTITION 202602 FROM it_spike_b.t; -ALTER TABLE it_spike_b.t_snap ATTACH PARTITION 202603 FROM it_spike_b.t; -ALTER TABLE it_spike_b.t_snap ATTACH PARTITION 202604 FROM it_spike_b.t; -SYSTEM FLUSH LOGS; -SELECT query_duration_ms, read_rows, written_rows, ProfileEvents['WriteBufferFromFileDescriptorWriteBytes'] AS bytes_written, substr(query, 1, 60) AS q FROM system.query_log WHERE type='QueryFinish' AND query LIKE 'ALTER TABLE it_spike_b.t_snap ATTACH PARTITION%' ORDER BY event_time_microseconds; - ┌─query_duration_ms─┬─read_rows─┬─written_rows─┬─bytes_written─┬─q────────────────────────────────────────────────────────────┐ -1. │ 32 │ 0 │ 0 │ 0 │ ALTER TABLE it_spike_b.t_snap ATTACH PARTITION 202601 FROM i │ -2. │ 32 │ 0 │ 0 │ 0 │ ALTER TABLE it_spike_b.t_snap ATTACH PARTITION 202602 FROM i │ -3. │ 27 │ 0 │ 0 │ 0 │ ALTER TABLE it_spike_b.t_snap ATTACH PARTITION 202603 FROM i │ -4. │ 16 │ 0 │ 0 │ 0 │ ALTER TABLE it_spike_b.t_snap ATTACH PARTITION 202604 FROM i │ - └───────────────────┴───────────┴──────────────┴───────────────┴──────────────────────────────────────────────────────────────┘ -SELECT count() FROM it_spike_b.t_snap; - ┌─count()─┐ -1. │ 0 │ - └─────────┘ --- hardlinks? compare inode of a data file in t and t_snap -SELECT table, name, path FROM system.parts WHERE database='it_spike_b' AND table IN ('t','t_snap') AND active AND partition='202601'; --- structure mismatches -CREATE TABLE it_spike_b.diff_order (id UInt64, ts DateTime, payload String) ENGINE = MergeTree PARTITION BY toYYYYMM(ts) ORDER BY (payload, id); -ALTER TABLE it_spike_b.diff_order ATTACH PARTITION 202601 FROM it_spike_b.t; -Received exception from server (version 26.6.1): -Code: 36. DB::Exception: Received from :9440. DB::Exception: Tables have different ordering. (BAD_ARGUMENTS) -(query: ALTER TABLE it_spike_b.diff_order ATTACH PARTITION 202601 FROM it_spike_b.t;) -CREATE TABLE it_spike_b.diff_partkey (id UInt64, ts DateTime, payload String) ENGINE = MergeTree PARTITION BY toYYYYMMDD(ts) ORDER BY id; -ALTER TABLE it_spike_b.diff_partkey ATTACH PARTITION 202601 FROM it_spike_b.t; -Received exception from server (version 26.6.1): -Code: 36. DB::Exception: Received from :9440. DB::Exception: Tables have different partition key. (BAD_ARGUMENTS) -(query: ALTER TABLE it_spike_b.diff_partkey ATTACH PARTITION 202601 FROM it_spike_b.t;) -CREATE TABLE it_spike_b.diff_coltype (id UInt64, ts DateTime, payload LowCardinality(String)) ENGINE = MergeTree PARTITION BY toYYYYMM(ts) ORDER BY id; -ALTER TABLE it_spike_b.diff_coltype ATTACH PARTITION 202601 FROM it_spike_b.t; -Received exception from server (version 26.6.1): -Code: 122. DB::Exception: Received from :9440. DB::Exception: Tables have different structure. (INCOMPATIBLE_COLUMNS) -(query: ALTER TABLE it_spike_b.diff_coltype ATTACH PARTITION 202601 FROM it_spike_b.t;) -CREATE TABLE it_spike_b.diff_colorder (id UInt64, payload String, ts DateTime) ENGINE = MergeTree PARTITION BY toYYYYMM(ts) ORDER BY id; -ALTER TABLE it_spike_b.diff_colorder ATTACH PARTITION 202601 FROM it_spike_b.t; -CREATE TABLE it_spike_b.diff_extracol (id UInt64, ts DateTime, payload String, extra UInt8 DEFAULT 0) ENGINE = MergeTree PARTITION BY toYYYYMM(ts) ORDER BY id; -ALTER TABLE it_spike_b.diff_extracol ATTACH PARTITION 202601 FROM it_spike_b.t; -Received exception from server (version 26.6.1): -Code: 122. DB::Exception: Received from :9440. DB::Exception: Tables have different structure. (INCOMPATIBLE_COLUMNS) -(query: ALTER TABLE it_spike_b.diff_extracol ATTACH PARTITION 202601 FROM it_spike_b.t;) -CREATE TABLE it_spike_b.diff_index (id UInt64, ts DateTime, payload String, INDEX ip payload TYPE bloom_filter GRANULARITY 1) ENGINE = MergeTree PARTITION BY toYYYYMM(ts) ORDER BY id; -ALTER TABLE it_spike_b.diff_index ATTACH PARTITION 202601 FROM it_spike_b.t; -CREATE TABLE it_spike_b.diff_settings (id UInt64, ts DateTime, payload String) ENGINE = MergeTree PARTITION BY toYYYYMM(ts) ORDER BY id SETTINGS index_granularity = 1024; -ALTER TABLE it_spike_b.diff_settings ATTACH PARTITION 202601 FROM it_spike_b.t; -CREATE TABLE it_spike_b.diff_engine (id UInt64, ts DateTime, payload String) ENGINE = ReplacingMergeTree PARTITION BY toYYYYMM(ts) ORDER BY id; -ALTER TABLE it_spike_b.diff_engine ATTACH PARTITION 202601 FROM it_spike_b.t; -CREATE TABLE it_spike_b.diff_ttl (id UInt64, ts DateTime, payload String) ENGINE = MergeTree PARTITION BY toYYYYMM(ts) ORDER BY id TTL ts + INTERVAL 10 YEAR; -ALTER TABLE it_spike_b.diff_ttl ATTACH PARTITION 202601 FROM it_spike_b.t; -CREATE TABLE it_spike_b.diff_codec (id UInt64 CODEC(Delta, ZSTD), ts DateTime, payload String) ENGINE = MergeTree PARTITION BY toYYYYMM(ts) ORDER BY id; -ALTER TABLE it_spike_b.diff_codec ATTACH PARTITION 202601 FROM it_spike_b.t; -SELECT 'diff_*' AS t, table, sum(rows) FROM system.parts WHERE database='it_spike_b' AND table LIKE 'diff_%' AND active GROUP BY table ORDER BY table; diff --git a/docs/design/spikes/2026-10-02-rebuild/cloud/b1_supplement.out b/docs/design/spikes/2026-10-02-rebuild/cloud/b1_supplement.out deleted file mode 100644 index e546ceb..0000000 --- a/docs/design/spikes/2026-10-02-rebuild/cloud/b1_supplement.out +++ /dev/null @@ -1,60 +0,0 @@ -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping --- Supplemental Cloud diagnostic after original preload failed with SETTING_CONSTRAINT_VIOLATION 452. --- Identical 5M-row SELECT without only the prohibited session setting; not part of original spike verdict. -INSERT INTO it_spike_b.t SELECT number, toDateTime('2026-01-01 00:00:00') + intDiv(number, 50000) * 3600 * 24, repeat('x', 40) FROM numbers(5000000); -ALTER TABLE it_spike_b.t_snap ATTACH PARTITION 202601 FROM it_spike_b.t; -ALTER TABLE it_spike_b.t_snap ATTACH PARTITION 202602 FROM it_spike_b.t; -ALTER TABLE it_spike_b.t_snap ATTACH PARTITION 202603 FROM it_spike_b.t; -ALTER TABLE it_spike_b.t_snap ATTACH PARTITION 202604 FROM it_spike_b.t; -SELECT count() AS attached_rows FROM it_spike_b.t_snap; - ┌─attached_rows─┐ -1. │ 5000000 │ -- 5.00 million - └───────────────┘ -ALTER TABLE it_spike_b.diff_order ATTACH PARTITION 202601 FROM it_spike_b.t; -Received exception from server (version 26.6.1): -Code: 36. DB::Exception: Received from :9440. DB::Exception: Tables have different ordering. (BAD_ARGUMENTS) -(query: ALTER TABLE it_spike_b.diff_order ATTACH PARTITION 202601 FROM it_spike_b.t;) -ALTER TABLE it_spike_b.diff_partkey ATTACH PARTITION 202601 FROM it_spike_b.t; -Received exception from server (version 26.6.1): -Code: 36. DB::Exception: Received from :9440. DB::Exception: Tables have different partition key. (BAD_ARGUMENTS) -(query: ALTER TABLE it_spike_b.diff_partkey ATTACH PARTITION 202601 FROM it_spike_b.t;) -ALTER TABLE it_spike_b.diff_coltype ATTACH PARTITION 202601 FROM it_spike_b.t; -Received exception from server (version 26.6.1): -Code: 122. DB::Exception: Received from :9440. DB::Exception: Tables have different structure. (INCOMPATIBLE_COLUMNS) -(query: ALTER TABLE it_spike_b.diff_coltype ATTACH PARTITION 202601 FROM it_spike_b.t;) -ALTER TABLE it_spike_b.diff_colorder ATTACH PARTITION 202601 FROM it_spike_b.t; -ALTER TABLE it_spike_b.diff_extracol ATTACH PARTITION 202601 FROM it_spike_b.t; -Received exception from server (version 26.6.1): -Code: 122. DB::Exception: Received from :9440. DB::Exception: Tables have different structure. (INCOMPATIBLE_COLUMNS) -(query: ALTER TABLE it_spike_b.diff_extracol ATTACH PARTITION 202601 FROM it_spike_b.t;) -ALTER TABLE it_spike_b.diff_index ATTACH PARTITION 202601 FROM it_spike_b.t; -ALTER TABLE it_spike_b.diff_settings ATTACH PARTITION 202601 FROM it_spike_b.t; -ALTER TABLE it_spike_b.diff_engine ATTACH PARTITION 202601 FROM it_spike_b.t; -ALTER TABLE it_spike_b.diff_ttl ATTACH PARTITION 202601 FROM it_spike_b.t; -ALTER TABLE it_spike_b.diff_codec ATTACH PARTITION 202601 FROM it_spike_b.t; -SELECT table,sum(rows) FROM system.parts WHERE database='it_spike_b' AND table LIKE 'diff_%' AND active GROUP BY table ORDER BY table; - ┌─table─────────┬─sum(rows)─┐ -1. │ diff_codec │ 1550000 │ -2. │ diff_colorder │ 1550000 │ -3. │ diff_engine │ 1550000 │ -4. │ diff_index │ 1550000 │ -5. │ diff_settings │ 1550000 │ -6. │ diff_ttl │ 1550000 │ - └───────────────┴───────────┘ -SELECT hostName(), count() FROM clusterAllReplicas('default', it_spike_b.t_snap) GROUP BY hostName() SETTINGS skip_unavailable_shards=1; - ┌─hostName()─────────────────────────┬─count()─┐ -1. │ replica-a │ 5000000 │ -2. │ replica-b │ 5000000 │ - └────────────────────────────────────┴─────────┘ diff --git a/docs/design/spikes/2026-10-02-rebuild/cloud/b2.out b/docs/design/spikes/2026-10-02-rebuild/cloud/b2.out deleted file mode 100644 index 3351ea6..0000000 --- a/docs/design/spikes/2026-10-02-rebuild/cloud/b2.out +++ /dev/null @@ -1,91 +0,0 @@ -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -SET send_logs_level = 'none'; --- Unpartitioned (PARTITION BY tuple()) -CREATE TABLE it_spike_b.u (id UInt64, v String) ENGINE = MergeTree ORDER BY id; -INSERT INTO it_spike_b.u SELECT number, toString(number) FROM numbers(1000); -INSERT INTO it_spike_b.u SELECT number, toString(number) FROM numbers(1000, 1000); -SELECT partition, partition_id, count() FROM system.parts WHERE database='it_spike_b' AND table='u' AND active GROUP BY ALL; - ┌─partition─┬─partition_id─┬─count()─┐ -1. │ tuple() │ all │ 2 │ - └───────────┴──────────────┴─────────┘ -CREATE TABLE it_spike_b.u_snap AS it_spike_b.u; -ALTER TABLE it_spike_b.u_snap ATTACH PARTITION tuple() FROM it_spike_b.u; -SELECT 'u_snap after tuple()', count() FROM it_spike_b.u_snap; - ┌─'u_snap after tuple()'─┬─count()─┐ -1. │ u_snap after tuple() │ 2000 │ - └────────────────────────┴─────────┘ -CREATE TABLE it_spike_b.u_snap2 AS it_spike_b.u; -ALTER TABLE it_spike_b.u_snap2 ATTACH PARTITION ID 'all' FROM it_spike_b.u; -SELECT 'u_snap2 after ID all', count() FROM it_spike_b.u_snap2; - ┌─'u_snap2 after ID all'─┬─count()─┐ -1. │ u_snap2 after ID all │ 2000 │ - └────────────────────────┴─────────┘ -CREATE TABLE it_spike_b.u_snap3 AS it_spike_b.u; -ALTER TABLE it_spike_b.u_snap3 ATTACH PARTITION ALL FROM it_spike_b.u; -SELECT 'u_snap3 after ALL', count() FROM it_spike_b.u_snap3; - ┌─'u_snap3 after ALL'─┬─count()─┐ -1. │ u_snap3 after ALL │ 2000 │ - └─────────────────────┴─────────┘ --- ALL on a partitioned table -CREATE TABLE it_spike_b.t_all AS it_spike_b.t; -ALTER TABLE it_spike_b.t_all ATTACH PARTITION ALL FROM it_spike_b.t; -SELECT 't_all after ATTACH PARTITION ALL', count() FROM it_spike_b.t_all; - ┌─'t_all after ATTACH PARTITION ALL'─┬─count()─┐ -1. │ t_all after ATTACH PARTITION ALL │ 0 │ - └────────────────────────────────────┴─────────┘ --- MOVE PARTITION TO TABLE (MergeTree -> MergeTree) -CREATE TABLE it_spike_b.u_stage AS it_spike_b.u; -INSERT INTO it_spike_b.u_stage SELECT number + 10000, 'staged' FROM numbers(500); -CREATE TABLE it_spike_b.u_dest AS it_spike_b.u; -INSERT INTO it_spike_b.u_dest SELECT number + 20000, 'mv-written' FROM numbers(7); -ALTER TABLE it_spike_b.u_stage MOVE PARTITION tuple() TO TABLE it_spike_b.u_dest; -SELECT 'after MOVE tuple()', (SELECT count() FROM it_spike_b.u_stage) AS stage, (SELECT count() FROM it_spike_b.u_dest) AS dest; - ┌─'after MOVE tuple()'─┬─stage─┬─dest─┐ -1. │ after MOVE tuple() │ 0 │ 507 │ - └──────────────────────┴───────┴──────┘ -INSERT INTO it_spike_b.u_stage SELECT number + 30000, 'staged2' FROM numbers(5); -ALTER TABLE it_spike_b.u_stage MOVE PARTITION ALL TO TABLE it_spike_b.u_dest; -Received exception from server (version 26.6.1): -Code: 344. DB::Exception: Received from :9440. DB::Exception: Only support DROP/DETACH/ATTACH PARTITION ALL currently. (SUPPORT_IS_DISABLED) -(query: ALTER TABLE it_spike_b.u_stage MOVE PARTITION ALL TO TABLE it_spike_b.u_dest;) -ALTER TABLE it_spike_b.u_stage MOVE PARTITION ID 'all' TO TABLE it_spike_b.u_dest; -SELECT 'after MOVE ID all', (SELECT count() FROM it_spike_b.u_stage) AS stage, (SELECT count() FROM it_spike_b.u_dest) AS dest; - ┌─'after MOVE ID all'─┬─stage─┬─dest─┐ -1. │ after MOVE ID all │ 0 │ 512 │ - └─────────────────────┴───────┴──────┘ --- MOVE between different engine families -CREATE TABLE it_spike_b.p_mt (id UInt64, ts DateTime, payload String) ENGINE = MergeTree PARTITION BY toYYYYMM(ts) ORDER BY id; -CREATE TABLE it_spike_b.p_rmt (id UInt64, ts DateTime, payload String) ENGINE = ReplacingMergeTree PARTITION BY toYYYYMM(ts) ORDER BY id; -INSERT INTO it_spike_b.p_mt VALUES (1, '2026-01-05', 'a'); -ALTER TABLE it_spike_b.p_mt MOVE PARTITION 202601 TO TABLE it_spike_b.p_rmt; -SELECT 'MT->RMT move', (SELECT count() FROM it_spike_b.p_mt), (SELECT count() FROM it_spike_b.p_rmt); - ┌─'MT->RMT move'─┬─_subquery_1─┬─_subquery_2─┐ -1. │ MT->RMT move │ 0 │ 1 │ - └────────────────┴─────────────┴─────────────┘ --- REPLACE PARTITION (idempotent overwrite) -CREATE TABLE it_spike_b.p_rep AS it_spike_b.p_mt; -INSERT INTO it_spike_b.p_mt VALUES (2, '2026-01-06', 'it_spike_b'); -ALTER TABLE it_spike_b.p_rep REPLACE PARTITION 202601 FROM it_spike_b.p_mt; -ALTER TABLE it_spike_b.p_rep REPLACE PARTITION 202601 FROM it_spike_b.p_mt; -SELECT 'REPLACE twice', count() FROM it_spike_b.p_rep; - ┌─'REPLACE twice'─┬─count()─┐ -1. │ REPLACE twice │ 1 │ - └─────────────────┴─────────┘ --- MOVE PARTITION to a table with different ORDER BY -CREATE TABLE it_spike_b.p_diff (id UInt64, ts DateTime, payload String) ENGINE = MergeTree PARTITION BY toYYYYMM(ts) ORDER BY (payload, id); -ALTER TABLE it_spike_b.p_mt MOVE PARTITION 202601 TO TABLE it_spike_b.p_diff; -Received exception from server (version 26.6.1): -Code: 36. DB::Exception: Received from :9440. DB::Exception: Tables have different ordering. (BAD_ARGUMENTS) -(query: ALTER TABLE it_spike_b.p_mt MOVE PARTITION 202601 TO TABLE it_spike_b.p_diff;) diff --git a/docs/design/spikes/2026-10-02-rebuild/cloud/b3.out b/docs/design/spikes/2026-10-02-rebuild/cloud/b3.out deleted file mode 100644 index b5796a0..0000000 --- a/docs/design/spikes/2026-10-02-rebuild/cloud/b3.out +++ /dev/null @@ -1,43 +0,0 @@ -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -CREATE DATABASE it_spike_b3; --- Exact original explicit Keeper-path engine, expected Cloud rejection; do not silently replace it. -CREATE TABLE it_spike_b3.explicit_t (id UInt64, ts DateTime, payload String) ENGINE = ReplicatedMergeTree('/clickhouse/tables/spike/t', 'r1') PARTITION BY toYYYYMM(ts) ORDER BY id; -Received exception from server (version 26.6.1): -Code: 80. DB::Exception: Received from :9440. DB::Exception: Explicit replica_name is specified in SharedMergeTree arguments. If you really want to specify it explicitly, then you should use some macros to distinguish different replicas. (INCORRECT_QUERY) -(query: -- Exact original explicit Keeper-path engine, expected Cloud rejection; do not silently replace it. -CREATE TABLE it_spike_b3.explicit_t (id UInt64, ts DateTime, payload String) ENGINE = ReplicatedMergeTree('/clickhouse/tables/spike/t', 'r1') PARTITION BY toYYYYMM(ts) ORDER BY id;) --- Meaningful SharedMergeTree analog of b3 ATTACH and MOVE (single Shared DB, two Cloud compute replicas). -CREATE TABLE it_spike_b3.t (id UInt64, ts DateTime, payload String) ENGINE = MergeTree PARTITION BY toYYYYMM(ts) ORDER BY id; -CREATE TABLE it_spike_b3.t_snap (id UInt64, ts DateTime, payload String) ENGINE = MergeTree PARTITION BY toYYYYMM(ts) ORDER BY id; -INSERT INTO it_spike_b3.t SELECT number, toDateTime('2026-01-01 00:00:00') + intDiv(number, 50000) * 3600 * 24, repeat('x', 40) FROM numbers(5000000); -ALTER TABLE it_spike_b3.t_snap ATTACH PARTITION 202601 FROM it_spike_b3.t; -ALTER TABLE it_spike_b3.t_snap ATTACH PARTITION 202602 FROM it_spike_b3.t; -ALTER TABLE it_spike_b3.t_snap ATTACH PARTITION 202603 FROM it_spike_b3.t; -ALTER TABLE it_spike_b3.t_snap ATTACH PARTITION 202604 FROM it_spike_b3.t; -SELECT hostName(), count() FROM clusterAllReplicas('default', it_spike_b3.t_snap) GROUP BY hostName() SETTINGS skip_unavailable_shards=1; - ┌─hostName()─────────────────────────┬─count()─┐ -1. │ replica-a │ 5000000 │ -2. │ replica-b │ 5000000 │ - └────────────────────────────────────┴─────────┘ -CREATE TABLE it_spike_b3.stage (id UInt64, ts DateTime, payload String) ENGINE = MergeTree PARTITION BY toYYYYMM(ts) ORDER BY (payload,id); -CREATE TABLE it_spike_b3.t_new (id UInt64, ts DateTime, payload String) ENGINE = MergeTree PARTITION BY toYYYYMM(ts) ORDER BY (payload,id); -INSERT INTO it_spike_b3.t_new VALUES (99999999, '2026-01-15', 'mv-row'); -INSERT INTO it_spike_b3.stage SELECT * FROM it_spike_b3.t_snap WHERE _partition_id = '202601'; -ALTER TABLE it_spike_b3.stage MOVE PARTITION 202601 TO TABLE it_spike_b3.t_new; -SELECT hostName(), count() FROM clusterAllReplicas('default', it_spike_b3.t_new) GROUP BY hostName() SETTINGS skip_unavailable_shards=1; - ┌─hostName()─────────────────────────┬─count()─┐ -1. │ replica-a │ 1550001 │ -2. │ replica-b │ 1550001 │ - └────────────────────────────────────┴─────────┘ diff --git a/docs/design/spikes/2026-10-02-rebuild/cloud/b_replicas.out b/docs/design/spikes/2026-10-02-rebuild/cloud/b_replicas.out deleted file mode 100644 index b544dce..0000000 --- a/docs/design/spikes/2026-10-02-rebuild/cloud/b_replicas.out +++ /dev/null @@ -1,15 +0,0 @@ -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -replica-a 2000 -replica-b 2000 diff --git a/docs/design/spikes/2026-10-02-rebuild/cloud/cleanup.out b/docs/design/spikes/2026-10-02-rebuild/cloud/cleanup.out deleted file mode 100644 index f9d3e65..0000000 --- a/docs/design/spikes/2026-10-02-rebuild/cloud/cleanup.out +++ /dev/null @@ -1,17 +0,0 @@ -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -password_file_deleted=yes -cleanup_exit=0 -remaining_spike_databases= [] -password_file_still_absent=yes diff --git a/docs/design/spikes/2026-10-02-rebuild/cloud/d4.out b/docs/design/spikes/2026-10-02-rebuild/cloud/d4.out deleted file mode 100644 index e879e6c..0000000 --- a/docs/design/spikes/2026-10-02-rebuild/cloud/d4.out +++ /dev/null @@ -1,269 +0,0 @@ -> DROP DATABASE IF EXISTS it_spike_d4 SYNC -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -> CREATE DATABASE it_spike_d4 -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -> CREATE TABLE it_spike_d4.t (id UInt64) ENGINE = MergeTree ORDER BY id -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -> CREATE TABLE it_spike_d4.t_new (id UInt64) ENGINE = MergeTree ORDER BY (intHash32(id), id) -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -> CREATE TABLE it_spike_d4.sink (id UInt64) ENGINE = MergeTree ORDER BY id -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -> CREATE MATERIALIZED VIEW it_spike_d4.mv_src TO it_spike_d4.sink AS SELECT id FROM it_spike_d4.t -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -> CREATE MATERIALIZED VIEW it_spike_d4.t_dual TO it_spike_d4.t_new AS SELECT id FROM it_spike_d4.t -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -> SELECT name, uuid FROM system.tables WHERE database='it_spike_d4' AND name IN ('t','t_new') ORDER BY name -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping - ┌─name──┬─uuid─────────────────────────────────┐ -1. │ t │ 0ee9158a-9199-4c8a-9a88-7b49b9a74a72 │ -2. │ t_new │ eac41d41-669c-4dc2-be17-319e1f9f68d7 │ - └───────┴──────────────────────────────────────┘ -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -> SELECT (SELECT count() FROM it_spike_d4.t) AS old_t_so_far -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping - ┌─old_t_so_far─┐ -1. │ 2000 │ - └──────────────┘ -> EXCHANGE TABLES it_spike_d4.t AND it_spike_d4.t_new -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -> DROP TABLE it_spike_d4.t_dual SYNC -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -> SELECT name, uuid FROM system.tables WHERE database='it_spike_d4' AND name IN ('t','t_new') ORDER BY name -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping - ┌─name──┬─uuid─────────────────────────────────┐ -1. │ t │ eac41d41-669c-4dc2-be17-319e1f9f68d7 │ -2. │ t_new │ 0ee9158a-9199-4c8a-9a88-7b49b9a74a72 │ - └───────┴──────────────────────────────────────┘ -> SYSTEM FLUSH LOGS -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -> SELECT query_duration_ms, substr(query,1,40) q FROM clusterAllReplicas('default', merge('system', '^query_log')) WHERE type='QueryFinish' AND (query LIKE 'EXCHANGE TABLES d4%' OR query LIKE 'DROP TABLE it_spike_d4.t_dual%') -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -> SELECT (SELECT count() FROM it_spike_d4.t) AS new_physical_named_t, (SELECT uniqExact(id) FROM it_spike_d4.t) AS new_t_uniq, (SELECT count() FROM it_spike_d4.t_new) AS old_physical_named_t_new, (SELECT count() FROM it_spike_d4.sink) AS sink_rows, (SELECT uniqExact(id) FROM it_spike_d4.sink) AS sink_uniq -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping - ┌─new_physical_named_t─┬─new_t_uniq─┬─old_physical_named_t_new─┬─sink_rows─┬─sink_uniq─┐ -1. │ 15000 │ 15000 │ 15000 │ 15000 │ 15000 │ - └──────────────────────┴────────────┴──────────────────────────┴───────────┴───────────┘ -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -5 EXCHANGE TABLES it_spike_d4.t AND it_spike_d4.t_new -62 EXCHANGE TABLES it_spike_d4.t AND it_spike_d4.t_new -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -replica-a QueryStart 0 DROP TABLE it_spike_d4.t_dual SYNC -replica-a QueryStart 0 DROP TABLE it_spike_d4.t_dual SYNC -replica-a QueryFinish 13017 DROP TABLE it_spike_d4.t_dual SYNC -replica-a QueryFinish 13026 DROP TABLE it_spike_d4.t_dual SYNC diff --git a/docs/design/spikes/2026-10-02-rebuild/cloud/d5.out b/docs/design/spikes/2026-10-02-rebuild/cloud/d5.out deleted file mode 100644 index c3edd9e..0000000 --- a/docs/design/spikes/2026-10-02-rebuild/cloud/d5.out +++ /dev/null @@ -1,308 +0,0 @@ -> DROP DATABASE IF EXISTS it_spike_d5 SYNC -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -> CREATE DATABASE it_spike_d5 -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -> CREATE TABLE it_spike_d5.t (id UInt64) ENGINE = MergeTree ORDER BY id -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -> CREATE TABLE it_spike_d5.t_new (id UInt64) ENGINE = MergeTree ORDER BY (intHash32(id), id) -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -> CREATE MATERIALIZED VIEW it_spike_d5.t_dual TO it_spike_d5.t_new AS SELECT id FROM it_spike_d5.t -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -> INSERT INTO it_spike_d5.t SETTINGS async_insert = 1, async_insert_use_adaptive_busy_timeout = 0, async_insert_busy_timeout_ms = 8000, async_insert_busy_timeout_max_ms = 8000, wait_for_async_insert = 0 VALUES (1),(2),(3) -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -> [background] INSERT INTO it_spike_d5.t SETTINGS async_insert = 1, async_insert_use_adaptive_busy_timeout = 0, async_insert_busy_timeout_ms = 8000, async_insert_busy_timeout_max_ms = 8000, wait_for_async_insert = 1 VALUES (4),(5) -> SELECT database, table, total_bytes, length(entries.query_id) entries FROM system.asynchronous_inserts WHERE database='it_spike_d5' -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping - ┌─database────┬─table─┬─total_bytes─┬─entries─┐ -1. │ it_spike_d5 │ t │ 7 │ 1 │ - └─────────────┴───────┴─────────────┴─────────┘ -> EXCHANGE TABLES it_spike_d5.t AND it_spike_d5.t_new -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -> SYSTEM FLUSH ASYNC INSERT QUEUE -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -Received exception from server (version 26.6.1): -Code: 497. DB::Exception: Received from :9440. DB::Exception: chm_it: Not enough privileges. To execute this query, it's necessary to have the grant SYSTEM FLUSH ASYNC INSERT QUEUE ON *.*. (ACCESS_DENIED) -(query: SYSTEM FLUSH ASYNC INSERT QUEUE) -> client of the wait_for_async_insert=1 insert saw: -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -Received exception from server (version 26.6.1): -Code: 741. DB::Exception: Received from :9440. DB::Exception: Table it_spike_d5.t (fe982bc0-f02b-4fd8-a6c7-f4f907d89014) does not match it_spike_d5.t_new (fe982bc0-f02b-4fd8-a6c7-f4f907d89014): While executing WaitForAsyncInsert. (TABLE_UUID_MISMATCH) -(query: INSERT INTO it_spike_d5.t SETTINGS async_insert = 1, async_insert_use_adaptive_busy_timeout = 0, async_insert_busy_timeout_ms = 8000, async_insert_busy_timeout_max_ms = 8000, wait_for_async_insert = 1 VALUES (4),(5)) -> SELECT (SELECT groupArray(id) FROM it_spike_d5.t) AS name_t_new_physical, (SELECT groupArray(id) FROM it_spike_d5.t_new) AS name_t_new_old_physical -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping - ┌─name_t_new_physical─┬─name_t_new_old_physical─┐ -1. │ [] │ [] │ - └─────────────────────┴─────────────────────────┘ -> SYSTEM FLUSH LOGS -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -> SELECT status, rows, substr(exception, 1, 110) AS exception FROM clusterAllReplicas('default', system.asynchronous_insert_log) WHERE database = 'it_spike_d5' ORDER BY event_time_microseconds -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping - ┌─status─────┬─rows─┬─exception──────────────────────────────────────────────────────────────────────────────────────────────────────┐ -1. │ FlushError │ 0 │ Code: 741. DB::Exception: Table it_spike_d5.t (fe982bc0-f02b-4fd8-a6c7-f4f907d89014) does not match it_spike_d │ - └────────────┴──────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ - -=== D5c: is an in-progress AsyncInsertFlush visible in system.processes? === -> CREATE TABLE it_spike_d5.slow_src (id UInt64) ENGINE = MergeTree ORDER BY id -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -> CREATE TABLE it_spike_d5.slow_dst (id UInt64) ENGINE = MergeTree ORDER BY id -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -> CREATE MATERIALIZED VIEW it_spike_d5.slow_mv TO it_spike_d5.slow_dst AS SELECT id FROM it_spike_d5.slow_src WHERE sleepEachRow(0.5) = 0 -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -> INSERT INTO it_spike_d5.slow_src SETTINGS async_insert = 1, async_insert_use_adaptive_busy_timeout = 0, async_insert_busy_timeout_ms = 8000, async_insert_busy_timeout_max_ms = 8000, wait_for_async_insert = 0 VALUES (1),(2),(3),(4),(5),(6) -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -Received exception from server (version 26.6.1): -Code: 497. DB::Exception: Received from :9440. DB::Exception: chm_it: Not enough privileges. To execute this query, it's necessary to have the grant SYSTEM FLUSH ASYNC INSERT QUEUE ON *.*. (ACCESS_DENIED) -(query: SYSTEM FLUSH ASYNC INSERT QUEUE) -> SELECT query_kind, round(elapsed, 2) AS elapsed, is_initial_query, substr(query, 1, 50) AS q FROM clusterAllReplicas('default', system.processes) WHERE query_kind != 'Select' -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping - ┌─query_kind─┬─elapsed─┬─is_initial_query─┬─q────────────────────────────────────────────────┐ -1. │ Drop │ 0.04 │ 1 │ DROP DATABASE IF EXISTS it_b471d47b0578f082 SYNC │ -2. │ Drop │ 0.04 │ 1 │ DROP DATABASE IF EXISTS it_b471d47b0578f082 SYNC │ -3. │ Insert │ 0.01 │ 1 │ INSERT INTO it_spike_b.p_mt VALUES │ - └────────────┴─────────┴──────────────────┴──────────────────────────────────────────────────┘ -> SELECT count() FROM it_spike_d5.slow_dst -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping - ┌─count()─┐ -1. │ 0 │ - └─────────┘ diff --git a/docs/design/spikes/2026-10-02-rebuild/cloud/e.out b/docs/design/spikes/2026-10-02-rebuild/cloud/e.out deleted file mode 100644 index e63e2be..0000000 --- a/docs/design/spikes/2026-10-02-rebuild/cloud/e.out +++ /dev/null @@ -1,89 +0,0 @@ -Traceback (most recent call last): - File "/private/tmp/chm-cloud-spike/e2e.py", line 259, in - rebuild(sys.argv[2]) - ~~~~~~~^^^^^^^^^^^^^ - File "/private/tmp/chm-cloud-spike/e2e.py", line 209, in rebuild - step5(c, fh) - ~~~~~^^^^^^^ - File "/private/tmp/chm-cloud-spike/e2e.py", line 186, in step5 - c.command(f"SYSTEM FLUSH ASYNC INSERT QUEUE {DB}.t") - ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - File "/Users/danyoung/.cache/uv/archive-v0/T5ttq3RJkygYl1C6/lib/python3.13/site-packages/clickhouse_connect/driver/_backendclient.py", line 147, in command - execution = self._backend.execute_command(bound_cmd, bind_params, data, external_data, runtime, transport_settings) - File "/Users/danyoung/.cache/uv/archive-v0/T5ttq3RJkygYl1C6/lib/python3.13/site-packages/clickhouse_connect/driver/_backend/http_sync.py", line 278, in execute_command - response = self.request( - plan.payload, - ...<5 lines>... - retryable=_read_request_retryable(runtime.retryable, plan.payload, plan.params, self.params, plan.form_files), - ) - File "/Users/danyoung/.cache/uv/archive-v0/T5ttq3RJkygYl1C6/lib/python3.13/site-packages/clickhouse_connect/driver/_backend/http_sync.py", line 398, in request - self.error_handler(response) - ~~~~~~~~~~~~~~~~~~^^^^^^^^^^ - File "/Users/danyoung/.cache/uv/archive-v0/T5ttq3RJkygYl1C6/lib/python3.13/site-packages/clickhouse_connect/driver/_backend/http_sync.py", line 129, in error_handler - raise build_http_error( - ...<6 lines>... - ) from None -clickhouse_connect.driver.exceptions.DatabaseError: Received ClickHouse exception, code: 497, server response: Code: 497. DB::Exception: chm_it: Not enough privileges. To execute this query, it's necessary to have the grant SYSTEM FLUSH ASYNC INSERT QUEUE ON *.*. (ACCESS_DENIED) (for url https://:8443) -rebuild exit=1 -=== ledger === -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -uuids recorded {"old": "bf568072-4c72-4ec1-81c8-cb2d0ef40f8a", "new": "9ab98492-3cc7-4cc2-a524-c80d7cea69fc"} -snapshot done {"202608": 2021306, "202607": 2019960, "202609": 2019361} -copy 202607 staged 2019960 -copy 202607 moved 2019960 -copy 202608 staged 2021306 -copy 202608 moved 2021306 -copy 202609 staged 2019361 -copy 202609 moved 2019361 -=== verify === -{ - "preload_rows": 6000000, - "preload_missing": 0, - "preload_duplicated": 0, - "writer_batches": 250, - "writer_rows_acked": { - "sync": 76000, - "async": 7800, - "async0": 34600 - }, - "writer_errors_logged": 0, - "missing_acked_ids_by_writer": {}, - "missing_sample": {}, - "duplicate_ids_inside_window": 0, - "duplicate_ids_outside_window": 0, - "outside_sample": [], - "final_rows": 6118400, - "final_distinct_ids": 6118400, - "dependent_mv_sink_ids_not_exactly_once": 0, - "sink_rows": 6118400, - "window_seconds": 4.692, - "step_marks_rel": [ - "1:start@0.000", - "1:end@0.887", - "2:start@1.077", - "2:end@3.193", - "3:start@3.193", - "3:end@4.692", - "4:start@4.692", - "4:stage-start:202607@5.304", - "4:moved:202607@6.787", - "4:stage-start:202608@6.986", - "4:moved:202608@8.917", - "4:stage-start:202609@9.057", - "4:moved:202609@10.980", - "4:end@10.980", - "5:start@10.980" - ] -} diff --git a/docs/design/spikes/2026-10-02-rebuild/cloud/e_noflush_cleanup.out b/docs/design/spikes/2026-10-02-rebuild/cloud/e_noflush_cleanup.out deleted file mode 100644 index f9d3e65..0000000 --- a/docs/design/spikes/2026-10-02-rebuild/cloud/e_noflush_cleanup.out +++ /dev/null @@ -1,17 +0,0 @@ -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -password_file_deleted=yes -cleanup_exit=0 -remaining_spike_databases= [] -password_file_still_absent=yes diff --git a/docs/design/spikes/2026-10-02-rebuild/cloud/e_noflush_kill.out b/docs/design/spikes/2026-10-02-rebuild/cloud/e_noflush_kill.out deleted file mode 100644 index 20d3e4f..0000000 --- a/docs/design/spikes/2026-10-02-rebuild/cloud/e_noflush_kill.out +++ /dev/null @@ -1,260 +0,0 @@ -[('202607', 2, 2000000), ('202608', 2, 2000000), ('202609', 2, 2000000)] -[14:26:00] SIGKILL rebuild python (moved=1, stage query written_rows=341905) -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -right after kill: server-side stage query still running? rebuild-stage-202608 3.81 358289 -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -ledger uuids recorded {"old": "631d2772-7c0a-4987-a407-886bfcb22f17", "new": "c17c13ce-5f39-4a75-8042-af7a32359b2c"} -ledger snapshot done {"202608": 2020106, "202607": 2019811, "202609": 2018083} -ledger copy 202607 staged 2019811 -ledger copy 202607 moved 2019811 -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -Received exception from server (version 26.6.1): -Code: 215. DB::Exception: Received from [cloud-host]:9440. DB::Exception: Column 'it_spike_e.t_stage._partition_id' is not under aggregate function and not in GROUP BY keys. In query SELECT 't_stage rows per partition', _partition_id, count() FROM it_spike_e.t_stage GROUP BY 1. (NOT_AN_AGGREGATE) -(query: SELECT 't_stage rows per partition', _partition_id, count() FROM it_spike_e.t_stage GROUP BY 1 FORMAT TSV) -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -after 2s: server-side stage query rebuild-stage-202608 6.79 653201 -[14:26:03] re-running rebuild -Traceback (most recent call last): - File "/private/tmp/chm-cloud-spike/e2e.py", line 259, in - rebuild(sys.argv[2]) - ~~~~~~~^^^^^^^^^^^^^ - File "/private/tmp/chm-cloud-spike/e2e.py", line 208, in rebuild - step4(c, fh) - ~~~~~^^^^^^^ - File "/private/tmp/chm-cloud-spike/e2e.py", line 170, in step4 - c.command( - ~~~~~~~~~^ - f"INSERT INTO {DB}.t_stage SELECT * FROM {DB}.t_snap WHERE _partition_id = '{p}'" + THROTTLE, - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - settings={"query_id": qid, "max_threads": 2, "max_insert_threads": 1, "max_block_size": 65536}, - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - ) - ^ - File "/Users/danyoung/.cache/uv/archive-v0/T5ttq3RJkygYl1C6/lib/python3.13/site-packages/clickhouse_connect/driver/_backendclient.py", line 147, in command - execution = self._backend.execute_command(bound_cmd, bind_params, data, external_data, runtime, transport_settings) - File "/Users/danyoung/.cache/uv/archive-v0/T5ttq3RJkygYl1C6/lib/python3.13/site-packages/clickhouse_connect/driver/_backend/http_sync.py", line 278, in execute_command - response = self.request( - plan.payload, - ...<5 lines>... - retryable=_read_request_retryable(runtime.retryable, plan.payload, plan.params, self.params, plan.form_files), - ) - File "/Users/danyoung/.cache/uv/archive-v0/T5ttq3RJkygYl1C6/lib/python3.13/site-packages/clickhouse_connect/driver/_backend/http_sync.py", line 398, in request - self.error_handler(response) - ~~~~~~~~~~~~~~~~~~^^^^^^^^^^ - File "/Users/danyoung/.cache/uv/archive-v0/T5ttq3RJkygYl1C6/lib/python3.13/site-packages/clickhouse_connect/driver/_backend/http_sync.py", line 129, in error_handler - raise build_http_error( - ...<6 lines>... - ) from None -clickhouse_connect.driver.exceptions.DatabaseError: Received ClickHouse exception, code: 394, server response: Code: 394. DB::Exception: Query was cancelled. (QUERY_WAS_CANCELLED) (for url https://[cloud-host]:8443) -[14:26:55] rebuild finished; writers keep going 3s -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -ledger uuids recorded {"old": "631d2772-7c0a-4987-a407-886bfcb22f17", "new": "c17c13ce-5f39-4a75-8042-af7a32359b2c"} -ledger snapshot done {"202608": 2020106, "202607": 2019811, "202609": 2018083} -ledger copy 202607 staged 2019811 -ledger copy 202607 moved 2019811 -ledger copy 202608 staged 2020106 -ledger copy 202608 moved 2020106 -ledger copy 202609 staged 2018083 -ledger copy 202609 moved 2018083 -ledger rebuild done -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -ledger SharedMergeTree 51cdf886-d096-44bf-8560-c64eab651382 -mv_dep MaterializedView 9b7ad2cd-1938-423c-80b9-52fa20f6c27a -sink SharedMergeTree 4d6b43c6-9e65-47e0-a57a-da6992b75f81 -t SharedMergeTree c17c13ce-5f39-4a75-8042-af7a32359b2c -t_old_rebuilt SharedMergeTree 631d2772-7c0a-4987-a407-886bfcb22f17 -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -final sorting key k, ts, id -{ - "preload_rows": 6000000, - "preload_missing": 0, - "preload_duplicated": 0, - "writer_batches": 1456, - "writer_rows_acked": { - "sync": 448000, - "async": 38600, - "async0": 207800 - }, - "writer_errors_logged": 0, - "missing_acked_ids_by_writer": { - "async0": 2200 - }, - "missing_sample": { - "async0": [ - 90185000, - 90185001, - 90185002 - ] - }, - "duplicate_ids_inside_window": 15800, - "duplicate_ids_outside_window": 0, - "outside_sample": [], - "final_rows": 6708000, - "final_distinct_ids": 6692200, - "dependent_mv_sink_ids_not_exactly_once": 0, - "sink_rows": 6692200, - "window_seconds": 3.224, - "step_marks_rel": [ - "1:start@0.000", - "1:end@0.723", - "2:start@0.826", - "2:end@2.331", - "3:start@2.331", - "3:end@3.224", - "4:start@3.224", - "4:stage-start:202607@3.573", - "4:moved:202607@25.364", - "4:stage-start:202608@25.504", - "1:start@33.452", - "1:end@33.626", - "2:start@33.811", - "2:end@33.846", - "3:start@33.846", - "3:skip@33.875", - "4:start@33.875", - "4:stage-start:202608@34.323", - "4:moved:202608@55.942", - "4:stage-start:202609@56.081", - "4:moved:202609@77.796", - "4:end@77.796", - "5:start@77.796", - "5:exchanged@77.909", - "5:t_dual_dropped@79.188", - "5:end@84.088" - ] -} -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -stage queries rebuild-stage-202607 QueryFinish 850 2019960 -stage queries rebuild-stage-202608 QueryFinish 1189 2021306 -stage queries rebuild-stage-202609 QueryFinish 1090 2019361 -stage queries rebuild-stage-202607 QueryFinish 722 2019612 -stage queries rebuild-stage-202608 QueryFinish 682 2020759 -stage queries rebuild-stage-202609 QueryFinish 758 2019049 -stage queries rebuild-stage-202607 QueryFinish 21273 2019811 -stage queries rebuild-stage-202608 ExceptionWhileProcessing 8632 833425 Code: 394. DB::Exception: Query was cancelled. (QUERY_WAS_CANCELLED) (version 26 -stage queries rebuild-stage-202608 QueryFinish 21102 2020106 -stage queries rebuild-stage-202609 QueryFinish 21195 2018083 -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -FlushError this run 12 36576 -wall_end_epoch=1791916031 diff --git a/docs/design/spikes/2026-10-02-rebuild/cloud/e_noflush_kill_diagnostics.out b/docs/design/spikes/2026-10-02-rebuild/cloud/e_noflush_kill_diagnostics.out deleted file mode 100644 index 6ec02d0..0000000 --- a/docs/design/spikes/2026-10-02-rebuild/cloud/e_noflush_kill_diagnostics.out +++ /dev/null @@ -1,39 +0,0 @@ -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -[replica-host] QueryStart 0 0 KILL QUERY WHERE query_id = \'rebuild-stage-202607\' SYNC -[replica-host] QueryFinish 0 2 KILL QUERY WHERE query_id = \'rebuild-stage-202607\' SYNC -[replica-host] QueryStart 0 0 KILL QUERY WHERE query_id = \'rebuild-stage-202608\' SYNC -[replica-host] QueryFinish 0 2 KILL QUERY WHERE query_id = \'rebuild-stage-202608\' SYNC -[replica-host] QueryStart 0 0 KILL QUERY WHERE query_id = \'rebuild-stage-202609\' SYNC -[replica-host] QueryFinish 0 2 KILL QUERY WHERE query_id = \'rebuild-stage-202609\' SYNC -[replica-host] QueryStart 0 0 KILL QUERY WHERE query_id = \'rebuild-stage-202607\' SYNC -[replica-host] QueryFinish 0 1 KILL QUERY WHERE query_id = \'rebuild-stage-202607\' SYNC -[replica-host] QueryStart 0 0 KILL QUERY WHERE query_id = \'rebuild-stage-202608\' SYNC -[replica-host] QueryFinish 0 2 KILL QUERY WHERE query_id = \'rebuild-stage-202608\' SYNC -[replica-host] QueryStart 0 0 KILL QUERY WHERE query_id = \'rebuild-stage-202608\' SYNC -[replica-host] QueryFinish 0 202 KILL QUERY WHERE query_id = \'rebuild-stage-202608\' SYNC -[replica-host] QueryStart 0 0 KILL QUERY WHERE query_id = \'rebuild-stage-202609\' SYNC -[replica-host] QueryFinish 0 2 KILL QUERY WHERE query_id = \'rebuild-stage-202609\' SYNC -[replica-host] QueryStart 0 0 KILL QUERY WHERE query_id = \'rebuild-stage-202607\' SYNC -[replica-host] QueryFinish 0 2 KILL QUERY WHERE query_id = \'rebuild-stage-202607\' SYNC -[replica-host] QueryStart 0 0 KILL QUERY WHERE query_id = \'rebuild-stage-202608\' SYNC -[replica-host] QueryFinish 0 2 KILL QUERY WHERE query_id = \'rebuild-stage-202608\' SYNC -[replica-host] QueryStart 0 0 KILL QUERY WHERE query_id = \'rebuild-stage-202608\' SYNC -[replica-host] QueryFinish 0 2 KILL QUERY WHERE query_id = \'rebuild-stage-202608\' SYNC -uuids recorded {"old": "ee510e37-5327-4b71-bb36-023fdc64cce1", "new": "61b0e8a4-767d-490e-9f68-4b287704c9cb"} -snapshot done {"202608": 2024237, "202607": 2023158, "202609": 2022135} -copy 202607 staged 2023158 -copy 202607 moved 2023158 -202608 4048474 -1 diff --git a/docs/design/spikes/2026-10-02-rebuild/cloud/e_noflush_kill_start.out b/docs/design/spikes/2026-10-02-rebuild/cloud/e_noflush_kill_start.out deleted file mode 100644 index 5a5be78..0000000 --- a/docs/design/spikes/2026-10-02-rebuild/cloud/e_noflush_kill_start.out +++ /dev/null @@ -1 +0,0 @@ -1791915918 diff --git a/docs/design/spikes/2026-10-02-rebuild/cloud/e_noflush_kill_true.out b/docs/design/spikes/2026-10-02-rebuild/cloud/e_noflush_kill_true.out deleted file mode 100644 index 0ad4d73..0000000 --- a/docs/design/spikes/2026-10-02-rebuild/cloud/e_noflush_kill_true.out +++ /dev/null @@ -1,224 +0,0 @@ -[('202607', 2, 2000000), ('202608', 2, 2000000), ('202609', 2, 2000000)] -[14:28:37] SIGKILL rebuild python (moved=1, stage query written_rows=325521) -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -right after kill: server-side stage query still running? rebuild-stage-202608 4.46 358289 -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -ledger uuids recorded {"old": "ee510e37-5327-4b71-bb36-023fdc64cce1", "new": "61b0e8a4-767d-490e-9f68-4b287704c9cb"} -ledger snapshot done {"202608": 2024237, "202607": 2023158, "202609": 2022135} -ledger copy 202607 staged 2023158 -ledger copy 202607 moved 2023158 -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -Received exception from server (version 26.6.1): -Code: 215. DB::Exception: Received from [cloud-host]:9440. DB::Exception: Column 'it_spike_e.t_stage._partition_id' is not under aggregate function and not in GROUP BY keys. In query SELECT 't_stage rows per partition', _partition_id, count() FROM it_spike_e.t_stage GROUP BY 1. (NOT_AN_AGGREGATE) -(query: SELECT 't_stage rows per partition', _partition_id, count() FROM it_spike_e.t_stage GROUP BY 1 FORMAT TSV) -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -after 2s: server-side stage query rebuild-stage-202608 7.46 653201 -[14:28:40] re-running rebuild -Traceback (most recent call last): - File "/private/tmp/chm-cloud-spike/e2e.py", line 259, in - rebuild(sys.argv[2]) - ~~~~~~~^^^^^^^^^^^^^ - File "/private/tmp/chm-cloud-spike/e2e.py", line 208, in rebuild - step4(c, fh) - ~~~~~^^^^^^^ - File "/private/tmp/chm-cloud-spike/e2e.py", line 175, in step4 - assert staged == expected, (p, staged, expected) - ^^^^^^^^^^^^^^^^^^ -AssertionError: ('202608', 4048474, 2024237) -[14:29:03] rebuild finished; writers keep going 3s -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -ledger uuids recorded {"old": "ee510e37-5327-4b71-bb36-023fdc64cce1", "new": "61b0e8a4-767d-490e-9f68-4b287704c9cb"} -ledger snapshot done {"202608": 2024237, "202607": 2023158, "202609": 2022135} -ledger copy 202607 staged 2023158 -ledger copy 202607 moved 2023158 -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -ledger SharedMergeTree c568de70-4005-4b90-b834-034c857e3840 -mv_dep MaterializedView 5b1c63a4-a966-4381-b1f1-d08e4ad765f9 -sink SharedMergeTree 53f12661-a78b-498d-ae59-2bfc7d1e7f95 -t SharedMergeTree ee510e37-5327-4b71-bb36-023fdc64cce1 -t_dual MaterializedView ce3ead4f-3b40-4dc4-9717-ff1e27bdd471 -t_new SharedMergeTree 61b0e8a4-767d-490e-9f68-4b287704c9cb -t_snap SharedMergeTree 2453d0d2-3440-494e-b8aa-4fd2edc000d5 -t_stage SharedMergeTree e117b8d9-21ec-444d-86db-4f9554a27671 -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -final sorting key id -{ - "preload_rows": 6000000, - "preload_missing": 0, - "preload_duplicated": 0, - "writer_batches": 1003, - "writer_rows_acked": { - "sync": 300000, - "async": 24200, - "async0": 146400 - }, - "writer_errors_logged": 0, - "missing_acked_ids_by_writer": {}, - "missing_sample": {}, - "duplicate_ids_inside_window": 0, - "duplicate_ids_outside_window": 0, - "outside_sample": [], - "final_rows": 6470600, - "final_distinct_ids": 6470600, - "dependent_mv_sink_ids_not_exactly_once": 0, - "sink_rows": 6470600, - "window_seconds": 6.286, - "step_marks_rel": [ - "1:start@0.000", - "1:end@1.639", - "2:start@1.834", - "2:end@3.903", - "3:start@3.903", - "3:end@6.286", - "4:start@6.286", - "4:stage-start:202607@6.789", - "4:moved:202607@30.081", - "4:stage-start:202608@30.228", - "1:start@39.011", - "1:end@39.193", - "2:start@39.312", - "2:end@39.368", - "3:start@39.368", - "3:skip@39.402", - "4:start@39.402", - "4:stage-start:202608@39.705" - ] -} -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -stage queries rebuild-stage-202607 QueryFinish 850 2019960 -stage queries rebuild-stage-202608 QueryFinish 1189 2021306 -stage queries rebuild-stage-202609 QueryFinish 1090 2019361 -stage queries rebuild-stage-202607 QueryFinish 722 2019612 -stage queries rebuild-stage-202608 QueryFinish 682 2020759 -stage queries rebuild-stage-202609 QueryFinish 758 2019049 -stage queries rebuild-stage-202607 QueryFinish 21273 2019811 -stage queries rebuild-stage-202608 ExceptionWhileProcessing 8632 833425 Code: 394. DB::Exception: Query was cancelled. (QUERY_WAS_CANCELLED) (version 26 -stage queries rebuild-stage-202608 QueryFinish 21102 2020106 -stage queries rebuild-stage-202609 QueryFinish 21195 2018083 -stage queries rebuild-stage-202607 QueryFinish 21878 2023158 -stage queries rebuild-stage-202608 QueryFinish 21916 2024237 -stage queries rebuild-stage-202608 QueryFinish 21307 2024237 -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -FlushError this run 12 36576 -wall_end_epoch=1791916157 diff --git a/docs/design/spikes/2026-10-02-rebuild/cloud/e_noflush_kill_true_start.out b/docs/design/spikes/2026-10-02-rebuild/cloud/e_noflush_kill_true_start.out deleted file mode 100644 index 1a86cfb..0000000 --- a/docs/design/spikes/2026-10-02-rebuild/cloud/e_noflush_kill_true_start.out +++ /dev/null @@ -1 +0,0 @@ -1791916058 diff --git a/docs/design/spikes/2026-10-02-rebuild/cloud/e_noflush_plain.out b/docs/design/spikes/2026-10-02-rebuild/cloud/e_noflush_plain.out deleted file mode 100644 index 114b618..0000000 --- a/docs/design/spikes/2026-10-02-rebuild/cloud/e_noflush_plain.out +++ /dev/null @@ -1,90 +0,0 @@ -[('202607', 2, 2000000), ('202608', 2, 2000000), ('202609', 2, 2000000)] -rebuild_exit=0 -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -uuids recorded -snapshot done -copy 202607 staged -copy 202607 moved -copy 202608 staged -copy 202608 moved -copy 202609 staged -copy 202609 moved -rebuild done -{ - "preload_rows": 6000000, - "preload_missing": 0, - "preload_duplicated": 0, - "writer_batches": 340, - "writer_rows_acked": { - "sync": 106000, - "async": 10200, - "async0": 47200 - }, - "writer_errors_logged": 0, - "missing_acked_ids_by_writer": { - "async0": 200 - }, - "missing_sample": { - "async0": [ - 90028200, - 90028201, - 90028202 - ] - }, - "duplicate_ids_inside_window": 19020, - "duplicate_ids_outside_window": 0, - "outside_sample": [], - "final_rows": 6182220, - "final_distinct_ids": 6163200, - "dependent_mv_sink_ids_not_exactly_once": 0, - "sink_rows": 6163200, - "window_seconds": 4.361, - "step_marks_rel": [ - "1:start@0.000", - "1:end@0.780", - "2:start@0.883", - "2:end@3.071", - "3:start@3.072", - "3:end@4.361", - "4:start@4.361", - "4:stage-start:202607@4.886", - "4:moved:202607@6.167", - "4:stage-start:202608@6.331", - "4:moved:202608@7.559", - "4:stage-start:202609@7.701", - "4:moved:202609@9.079", - "4:end@9.079", - "5:start@9.079", - "5:exchanged@9.196", - "5:t_dual_dropped@10.768", - "5:end@14.848" - ] -} -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -Ok 450 86816 -FlushError 1 0 -wall_end_epoch=1791915912 diff --git a/docs/design/spikes/2026-10-02-rebuild/cloud/e_noflush_start.out b/docs/design/spikes/2026-10-02-rebuild/cloud/e_noflush_start.out deleted file mode 100644 index 23d7c45..0000000 --- a/docs/design/spikes/2026-10-02-rebuild/cloud/e_noflush_start.out +++ /dev/null @@ -1 +0,0 @@ -1791915875 diff --git a/docs/design/spikes/2026-10-02-rebuild/cloud/e_setup.out b/docs/design/spikes/2026-10-02-rebuild/cloud/e_setup.out deleted file mode 100644 index 7277114..0000000 --- a/docs/design/spikes/2026-10-02-rebuild/cloud/e_setup.out +++ /dev/null @@ -1 +0,0 @@ -[('202607', 2, 2000000), ('202608', 2, 2000000), ('202609', 2, 2000000)] diff --git a/docs/design/spikes/2026-10-02-rebuild/cloud/metadata.out b/docs/design/spikes/2026-10-02-rebuild/cloud/metadata.out deleted file mode 100644 index 458d412..0000000 --- a/docs/design/spikes/2026-10-02-rebuild/cloud/metadata.out +++ /dev/null @@ -1,26 +0,0 @@ -Unknown setting 'dynamic_disk_allow_from_env', skipping -Unknown setting 'dynamic_disk_allow_include', skipping -Unknown setting 'dynamic_disk_allow_from_zk', skipping -Unknown setting 's3_allow_server_credentials_in_user_queries', skipping -Unknown setting 'execute_exists_as_scalar_subquery', skipping -Unknown setting 'distributed_cache_write_request_max_tries', skipping -Unknown setting 'show_data_lake_catalogs_in_system_tables', skipping -Unknown setting 'enable_producing_buckets_out_of_order_in_aggregation', skipping -Unknown setting 'enable_full_text_index', skipping -Unknown setting 'use_text_index_tokens_cache', skipping -Unknown setting 'use_text_index_header_cache', skipping -Unknown setting 'type_json_allow_duplicated_key_with_literal_and_nested_object', skipping -Unknown setting 'max_dynamic_subcolumns_in_json_type_parsing', skipping -t SharedMergeTree -t_snap SharedMergeTree -stage SharedMergeTree -t SharedMergeTree -t_new SharedMergeTree -t_snap SharedMergeTree -t SharedMergeTree -t_new SharedMergeTree -t_snap SharedMergeTree -replica-a 26.6.1.2326 -replica-b 26.6.1.2326 -scratch_created_epoch=1791905805 -1791905970 diff --git a/docs/design/spikes/2026-10-02-rebuild/reproduce_client.py b/docs/design/spikes/2026-10-02-rebuild/reproduce_client.py deleted file mode 100644 index d4f9bd5..0000000 --- a/docs/design/spikes/2026-10-02-rebuild/reproduce_client.py +++ /dev/null @@ -1,45 +0,0 @@ -"""Run archived clickhouse-client SQL through the fixture's local HTTP endpoint.""" - -import argparse -import sys -from pathlib import Path - -import clickhouse_connect - -from ch_migrate.config import get_env_config - - -def main(): - parser = argparse.ArgumentParser() - parser.add_argument("-q", "--query", required=True) - parser.add_argument("--query_id") - parser.add_argument("--format", default="TabSeparated") - args = parser.parse_args() - config = get_env_config("it", Path("config.yaml")) - if config["host"] not in ("127.0.0.1", "localhost"): - raise SystemExit("This spike adapter only accepts an owned loopback fixture") - client = clickhouse_connect.get_client( - host=config["host"], - port=config["port"], - username=config["migration_user"], - password=config["password"], - secure=config.get("secure", False), - send_receive_timeout=60, - ) - try: - settings = {"query_id": args.query_id} if args.query_id else None - reads_result = args.query.lstrip().split(None, 1)[0].upper() in { - "SELECT", - "WITH", - "SHOW", - "DESCRIBE", - "EXPLAIN", - } - output_format = args.format if reads_result else None - sys.stdout.buffer.write(client.raw_query(args.query, settings=settings, fmt=output_format)) - finally: - client.close() - - -if __name__ == "__main__": - main() diff --git a/docs/design/spikes/2026-10-02-waiting/resume_fence.py b/docs/design/spikes/2026-10-02-waiting/resume_fence.py index 98580ec..668a9cd 100644 --- a/docs/design/spikes/2026-10-02-waiting/resume_fence.py +++ b/docs/design/spikes/2026-10-02-waiting/resume_fence.py @@ -3,8 +3,8 @@ import time from pathlib import Path -from ch_migrate.config import get_env_config -from ch_migrate.connection import get_client +from clickhouse_alembic.config import get_env_config +from clickhouse_alembic.connection import get_client config = get_env_config("it", Path("config.yaml")) client = get_client(config) diff --git a/docs/design/spikes/2026-10-02-waiting/submit_without_ack.py b/docs/design/spikes/2026-10-02-waiting/submit_without_ack.py index bf51061..c8d2742 100644 --- a/docs/design/spikes/2026-10-02-waiting/submit_without_ack.py +++ b/docs/design/spikes/2026-10-02-waiting/submit_without_ack.py @@ -6,7 +6,7 @@ from pathlib import Path from urllib.parse import urlencode -from ch_migrate.config import get_env_config +from clickhouse_alembic.config import get_env_config config = get_env_config("it", Path("config.yaml")) token = sys.argv[1] diff --git a/docs/design/spikes/2026-10-13-exchange-race/probe.py b/docs/design/spikes/2026-10-13-exchange-race/probe.py deleted file mode 100644 index 178b963..0000000 --- a/docs/design/spikes/2026-10-13-exchange-race/probe.py +++ /dev/null @@ -1,111 +0,0 @@ -"""Does an insert racing EXCHANGE land only in the old storage (silently lost)? - -Per trial: src (live `events`) with a dual MV src -> dst; writers insert continuously; -EXCHANGE events <-> dst; then every acknowledged id must be in the live `events`. -Also checks whether part_log identifies the stranded rows: NewPart in the old storage -after the swap from a query that made no part in the new storage. -""" - -import sys -import threading -import time - -import clickhouse_connect - -PORT, MODE, TRIALS = int(sys.argv[1]), sys.argv[2], int(sys.argv[3]) -ASYNC = { - "async_insert": 1, - "async_insert_use_adaptive_busy_timeout": 0, - "async_insert_busy_timeout_ms": 5, - "async_insert_busy_timeout_max_ms": 5, -} -SETTINGS = { - "sync": {"async_insert": 0}, - "async0": {**ASYNC, "wait_for_async_insert": 0}, - "async1": {**ASYNC, "wait_for_async_insert": 1}, -}[MODE] - - -def client(): - if PORT == 0: # Cloud: CH_MIGRATE_IT_URL from the redacting runner. - import os - from urllib.parse import unquote, urlsplit - - url = urlsplit(os.environ["CH_MIGRATE_IT_URL"]) - return clickhouse_connect.get_client( - host=url.hostname, port=url.port, username=url.username, - password=unquote(url.password), secure=True, - ) - return clickhouse_connect.get_client(host="127.0.0.1", port=PORT, password="probe") - - -def trial(admin, number): - db = f"it_race_{number}" - admin.command(f"DROP DATABASE IF EXISTS {db} SYNC") - admin.command(f"CREATE DATABASE {db}") - for name in ("events", "dst"): - admin.command(f"CREATE TABLE {db}.{name} (id UInt64) ENGINE = MergeTree ORDER BY id") - admin.command(f"CREATE MATERIALIZED VIEW {db}.dual TO {db}.dst AS SELECT id FROM {db}.events") - uuids = dict( - admin.query( - f"SELECT name, toString(uuid) FROM system.tables WHERE database = '{db}'" - ).result_rows - ) - acked, errors, stop = [], [], threading.Event() - - def write(offset): - own = client() - next_id = offset - while not stop.is_set(): - try: - own.command(f"INSERT INTO {db}.events VALUES ({next_id})", settings=SETTINGS) - acked.append(next_id) - except Exception as exc: # 741 for async1 is an honest rejection. - errors.append(str(exc)[:80]) - next_id += 8 - - threads = [threading.Thread(target=write, args=(k,)) for k in range(8)] - for thread in threads: - thread.start() - time.sleep(0.15) - swap_start = admin.command("SELECT toString(now64(6))") - admin.command(f"EXCHANGE TABLES {db}.events AND {db}.dst") - time.sleep(0.15) - stop.set() - for thread in threads: - thread.join() - time.sleep(0.3) # Let 5 ms async timers fire. - consistent = {"select_sequential_consistency": 1} - live = {r[0] for r in admin.query(f"SELECT id FROM {db}.events", settings=consistent).result_rows} - old = {r[0] for r in admin.query(f"SELECT id FROM {db}.dst", settings=consistent).result_rows} - lost = set(acked) - live - in_old_only = lost & old - part_log = "clusterAllReplicas('default', system.part_log)" if PORT == 0 else "system.part_log" - admin.command("SYSTEM FLUSH LOGS" + (" ON CLUSTER 'default'" if PORT == 0 else "")) - stranded = admin.command( - f"SELECT sum(rows) FROM {part_log} WHERE event_type = 'NewPart' " - f"AND table_uuid = '{uuids['events']}' AND event_time_microseconds >= '{swap_start}' " - f"AND query_id NOT IN (SELECT query_id FROM {part_log} WHERE event_type = 'NewPart' " - f"AND table_uuid = '{uuids['dst']}')" - ) - if int(stranded or 0) != len(in_old_only): - print("MISMATCH trial", number, "stranded", stranded, "in_old_only", len(in_old_only)) - admin.command(f"DROP DATABASE {db} SYNC") - return len(acked), len(lost), len(in_old_only), len(errors), errors[:1] - - -admin = client() -totals = [0, 0, 0, 0] -hits = 0 -for number in range(TRIALS): - *counts, sample = trial(admin, number) - totals = [a + b for a, b in zip(totals, counts)] - if counts[1]: - hits += 1 - print("trial", number, "acked", counts[0], "lost", counts[1], "in_old_only", counts[2]) - if sample and number < 3: - print("error sample", sample) -print( - MODE, "trials", TRIALS, "with_loss", hits, "acked", totals[0], "lost", totals[1], - "in_old_only", totals[2], "errors", totals[3], -) diff --git a/docs/examples/github-actions-drift.yml b/docs/examples/github-actions-drift.yml deleted file mode 100644 index f4cd89a..0000000 --- a/docs/examples/github-actions-drift.yml +++ /dev/null @@ -1,85 +0,0 @@ -name: ClickHouse schema drift - -on: - schedule: - - cron: '17 6 * * *' - pull_request: - -permissions: - contents: read - -jobs: - drift: - # Fork pull requests cannot access repository secrets. Never use - # pull_request_target to expose database credentials to untrusted code. - if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository - runs-on: ubuntu-latest - env: - # Set to an approved ch-migrate-cli requirement with --json support. - # For example, pin a released version after that feature is published. - # There is no fallback to an older release without the required command. - CH_MIGRATE_PACKAGE: ${{ vars.CH_MIGRATE_PACKAGE }} - CH_MIGRATE_PYTHON: '3.12' - # These name a config.yaml environment and its committed snapshot folder. - CH_MIGRATE_ENV: ${{ vars.CH_MIGRATE_ENV }} - CH_MIGRATE_SNAPSHOT_DIR: ${{ vars.CH_MIGRATE_SNAPSHOT_DIR }} - UV_TOOL_DIR: ${{ github.workspace }}/.ch-migrate-tools - UV_TOOL_BIN_DIR: ${{ github.workspace }}/.ch-migrate-bin - steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 - with: - persist-credentials: false - - uses: astral-sh/setup-uv@c18668ad3cf93ea998bef934396af7bb5c839dc7 # v10.2.0 - with: - python-version: '3.12' - - name: Install ch-migrate - id: install - run: | - : "${CH_MIGRATE_PACKAGE:?Set a package requirement with JSON support}" - : "${CH_MIGRATE_ENV:?Set the config.yaml environment name}" - : "${CH_MIGRATE_SNAPSHOT_DIR:?Set the committed snapshot path}" - uv tool install --python "$CH_MIGRATE_PYTHON" "$CH_MIGRATE_PACKAGE" - - name: Write scoped database credential - id: credentials - env: - # Password for config.yaml's migration_user in CH_MIGRATE_ENV. - # Use a dedicated schema-inspection account, not an admin credential. - DRIFT_MIGRATION_PASSWORD: ${{ secrets.DRIFT_MIGRATION_PASSWORD }} - run: | - umask 077 - python - <<'PY' - import json - import os - from pathlib import Path - - password = os.environ['DRIFT_MIGRATION_PASSWORD'] - if not password: - raise SystemExit('DRIFT_MIGRATION_PASSWORD is not configured') - key = 'CH_' + os.environ['CH_MIGRATE_ENV'].upper() + '_MIGRATION_PASSWORD' - Path('.env.local').write_text(key + '=' + json.dumps(password) + '\n') - PY - - name: Compare committed schema with live schema - id: compare - continue-on-error: true - run: | - "$UV_TOOL_BIN_DIR/ch-migrate" diff "$CH_MIGRATE_ENV" \ - --snapshot-dir "$CH_MIGRATE_SNAPSHOT_DIR" --json > drift.json - - name: Preserve drift report even when comparison fails - if: always() - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 - with: - name: schema-drift - path: drift.json - if-no-files-found: warn - - name: Fail on drift or comparison errors - id: gate - if: always() - env: - DRIFT_OUTCOME: ${{ steps.compare.outcome }} - run: | - # compare exits 1 on drift and 2 on errors; neither is a clean result. - test "$DRIFT_OUTCOME" = success - - name: Remove the temporary credential file - id: cleanup - if: always() - run: rm -f .env.local diff --git a/docs/schemas/diff.schema.json b/docs/schemas/diff.schema.json deleted file mode 100644 index 19d79f1..0000000 --- a/docs/schemas/diff.schema.json +++ /dev/null @@ -1,102 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "diff.schema.json", - "title": "ch-migrate diff schema version 1", - "type": "object", - "required": [ - "schema_version", - "command" - ], - "properties": { - "schema_version": { - "const": 1 - }, - "command": { - "const": "diff" - }, - "error": { - "type": "string" - }, - "objects": { - "type": "array", - "items": { - "type": "object", - "properties": { - "type": { - "enum": [ - "table", - "view", - "materialized_view", - "dictionary" - ] - }, - "name": { - "type": "string" - }, - "status": { - "enum": [ - "in_sync", - "modified", - "local_only", - "remote_only" - ] - }, - "details": { - "type": "array", - "items": { - "type": "object", - "properties": { - "field": { - "type": "string" - }, - "local": { - "type": [ - "string", - "null" - ] - }, - "remote": { - "type": [ - "string", - "null" - ] - }, - "message": { - "type": "string" - } - }, - "required": [ - "field", - "local", - "remote", - "message" - ] - } - } - }, - "required": [ - "type", - "name", - "status", - "details" - ] - } - }, - "in_sync": { - "type": "boolean" - } - }, - "anyOf": [ - { - "required": [ - "error" - ] - }, - { - "required": [ - "objects", - "in_sync" - ] - } - ] -} diff --git a/docs/schemas/history.schema.json b/docs/schemas/history.schema.json deleted file mode 100644 index 75db349..0000000 --- a/docs/schemas/history.schema.json +++ /dev/null @@ -1,87 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "history.schema.json", - "title": "ch-migrate history schema version 1", - "type": "object", - "required": [ - "schema_version", - "command" - ], - "properties": { - "schema_version": { - "const": 1 - }, - "command": { - "const": "history" - }, - "error": { - "type": "string" - }, - "revisions": { - "type": "array", - "items": { - "type": "object", - "properties": { - "revision": { - "type": "string" - }, - "down_revisions": { - "type": "array", - "items": { - "type": "string" - }, - "uniqueItems": true - }, - "description": { - "type": [ - "string", - "null" - ] - }, - "create_date": { - "type": [ - "string", - "null" - ] - }, - "path": { - "type": "string" - }, - "applied": { - "type": [ - "boolean", - "null" - ] - }, - "irreversible": { - "type": [ - "string", - "null" - ] - } - }, - "required": [ - "revision", - "down_revisions", - "description", - "create_date", - "path", - "applied", - "irreversible" - ] - } - } - }, - "anyOf": [ - { - "required": [ - "error" - ] - }, - { - "required": [ - "revisions" - ] - } - ] -} diff --git a/docs/schemas/lint.schema.json b/docs/schemas/lint.schema.json deleted file mode 100644 index 8bea30c..0000000 --- a/docs/schemas/lint.schema.json +++ /dev/null @@ -1,124 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "lint.schema.json", - "title": "ch-migrate lint schema version 1", - "type": "object", - "required": [ - "schema_version", - "command" - ], - "properties": { - "schema_version": { - "const": 1 - }, - "command": { - "const": "lint" - }, - "error": { - "type": "string" - }, - "findings": { - "type": "array", - "items": { - "type": "object", - "properties": { - "rule": { - "type": "string" - }, - "severity": { - "enum": [ - "error", - "warn", - "info", - "off" - ] - }, - "blocking": { - "type": "boolean" - }, - "file": { - "type": [ - "string", - "null" - ] - }, - "line": { - "type": [ - "integer", - "null" - ], - "minimum": 1 - }, - "message": { - "type": "string" - }, - "waived": { - "type": [ - "string", - "null" - ] - } - }, - "required": [ - "rule", - "severity", - "blocking", - "file", - "line", - "message", - "waived" - ] - } - }, - "counts": { - "type": "object", - "properties": { - "total": { - "type": "integer", - "minimum": 0 - }, - "error": { - "type": "integer", - "minimum": 0 - }, - "warning": { - "type": "integer", - "minimum": 0 - }, - "info": { - "type": "integer", - "minimum": 0 - }, - "blocking": { - "type": "integer", - "minimum": 0 - }, - "waived": { - "type": "integer", - "minimum": 0 - } - }, - "required": [ - "total", - "error", - "warning", - "info", - "blocking", - "waived" - ] - } - }, - "anyOf": [ - { - "required": [ - "error" - ] - }, - { - "required": [ - "findings", - "counts" - ] - } - ] -} diff --git a/docs/schemas/plan.schema.json b/docs/schemas/plan.schema.json deleted file mode 100644 index 045d7eb..0000000 --- a/docs/schemas/plan.schema.json +++ /dev/null @@ -1,427 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "plan.schema.json", - "title": "ch-migrate plan schema version 1", - "type": "object", - "required": [ - "schema_version", - "command" - ], - "properties": { - "schema_version": { - "const": 1 - }, - "command": { - "const": "plan" - }, - "error": { - "type": "string" - }, - "database": { - "type": "string" - }, - "environment": { - "type": "string" - }, - "gate_would_refuse": { - "type": "boolean" - }, - "warnings": { - "type": "array", - "items": { - "type": "string" - } - }, - "findings": { - "type": "array", - "items": { - "type": "object", - "properties": { - "rule": { - "type": "string" - }, - "severity": { - "enum": [ - "error", - "warn", - "info", - "off" - ] - }, - "blocking": { - "type": "boolean" - }, - "file": { - "type": [ - "string", - "null" - ] - }, - "line": { - "type": [ - "integer", - "null" - ], - "minimum": 1 - }, - "message": { - "type": "string" - }, - "waived": { - "type": [ - "string", - "null" - ] - } - }, - "required": [ - "rule", - "severity", - "blocking", - "file", - "line", - "message", - "waived" - ] - } - }, - "counts": { - "type": "object", - "properties": { - "total": { - "type": "integer", - "minimum": 0 - }, - "error": { - "type": "integer", - "minimum": 0 - }, - "warning": { - "type": "integer", - "minimum": 0 - }, - "info": { - "type": "integer", - "minimum": 0 - }, - "blocking": { - "type": "integer", - "minimum": 0 - }, - "waived": { - "type": "integer", - "minimum": 0 - } - }, - "required": [ - "total", - "error", - "warning", - "info", - "blocking", - "waived" - ] - }, - "migrations": { - "type": "array", - "items": { - "type": "object", - "properties": { - "revision": { - "type": "string" - }, - "path": { - "type": "string" - }, - "statements": { - "type": "array", - "items": { - "type": "object", - "properties": { - "sql": { - "type": "string" - }, - "suggested_command": { - "type": "string", - "minLength": 1 - }, - "file": { - "type": "string" - }, - "line": { - "type": "integer", - "minimum": 1 - }, - "classification": { - "type": "object", - "properties": { - "kind": { - "enum": [ - "metadata", - "mutation", - "rebuild", - "other" - ] - }, - "table": { - "type": [ - "string", - "null" - ] - }, - "detail": { - "type": "string" - } - }, - "required": [ - "kind", - "table", - "detail" - ] - }, - "size": { - "anyOf": [ - { - "type": "null" - }, - { - "type": "object", - "properties": { - "precision": { - "enum": [ - "exact", - "ceiling" - ] - }, - "compressed_bytes": { - "type": "integer", - "minimum": 0 - }, - "uncompressed_bytes": { - "type": "integer", - "minimum": 0 - }, - "part_count": { - "type": "integer", - "minimum": 0 - }, - "columns": { - "type": "array", - "items": { - "type": "string" - } - }, - "partition_id": { - "type": [ - "string", - "null" - ] - } - }, - "required": [ - "precision", - "compressed_bytes", - "uncompressed_bytes", - "part_count", - "columns", - "partition_id" - ] - } - ] - }, - "downstream": { - "type": "array", - "items": { - "type": "object", - "properties": { - "name": { - "type": "string" - }, - "type": { - "enum": [ - "materialized_view", - "dictionary" - ] - } - }, - "required": [ - "name", - "type" - ] - } - }, - "findings": { - "type": "array", - "items": { - "type": "object", - "properties": { - "rule": { - "type": "string" - }, - "severity": { - "enum": [ - "error", - "warn", - "info", - "off" - ] - }, - "blocking": { - "type": "boolean" - }, - "file": { - "type": [ - "string", - "null" - ] - }, - "line": { - "type": [ - "integer", - "null" - ], - "minimum": 1 - }, - "message": { - "type": "string" - }, - "waived": { - "type": [ - "string", - "null" - ] - } - }, - "required": [ - "rule", - "severity", - "blocking", - "file", - "line", - "message", - "waived" - ] - } - }, - "rebuild": { - "anyOf": [ - { - "type": "null" - }, - { - "type": "object", - "properties": { - "findings": { - "type": "array", - "items": { - "type": "object", - "properties": { - "code": { - "type": "string" - }, - "severity": { - "enum": [ - "refusal", - "warning" - ] - }, - "message": { - "type": "string" - }, - "details": { - "type": "object" - } - }, - "required": [ - "code", - "severity", - "message", - "details" - ] - } - }, - "bytes_on_disk": { - "type": "integer", - "minimum": 0 - }, - "part_count": { - "type": "integer", - "minimum": 0 - }, - "partition_parts": { - "type": "object", - "additionalProperties": { - "type": "integer", - "minimum": 0 - } - }, - "insert_rows_per_second": { - "type": "number", - "minimum": 0 - }, - "engine_note": { - "type": "string" - } - }, - "required": [ - "findings", - "bytes_on_disk", - "part_count", - "partition_parts", - "insert_rows_per_second", - "engine_note" - ] - } - ] - } - }, - "required": [ - "sql", - "file", - "line", - "classification", - "size", - "downstream", - "findings", - "rebuild" - ] - } - } - }, - "required": [ - "revision", - "path", - "statements" - ] - } - } - }, - "oneOf": [ - { - "required": [ - "error" - ], - "not": { - "required": [ - "migrations" - ] - } - }, - { - "required": [ - "database", - "environment", - "gate_would_refuse", - "warnings", - "findings", - "counts", - "migrations" - ], - "not": { - "required": [ - "error" - ] - } - } - ] -} diff --git a/docs/schemas/status.schema.json b/docs/schemas/status.schema.json deleted file mode 100644 index 5e5d6c8..0000000 --- a/docs/schemas/status.schema.json +++ /dev/null @@ -1,72 +0,0 @@ -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "status.schema.json", - "title": "ch-migrate status schema version 1", - "type": "object", - "required": [ - "schema_version", - "command" - ], - "properties": { - "schema_version": { - "const": 1 - }, - "command": { - "const": "status" - }, - "error": { - "type": "string" - }, - "database": { - "type": "string" - }, - "current_heads": { - "type": "array", - "items": { - "type": "string" - }, - "uniqueItems": true - }, - "script_heads": { - "type": "array", - "items": { - "type": "string" - }, - "uniqueItems": true - }, - "pending": { - "type": "array", - "items": { - "type": "string" - }, - "uniqueItems": true - }, - "applied": { - "type": "array", - "items": { - "type": "string" - }, - "uniqueItems": true - }, - "at_head": { - "type": "boolean" - } - }, - "anyOf": [ - { - "required": [ - "error" - ] - }, - { - "required": [ - "database", - "current_heads", - "script_heads", - "pending", - "applied", - "at_head" - ] - } - ] -} diff --git a/pyproject.toml b/pyproject.toml index 609dff9..389d5ee 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -50,7 +50,6 @@ dev = [ "isort>=5.12.0", "mypy>=1.5.0", "types-PyYAML", - "jsonschema>=4.18", ] [build-system] @@ -75,7 +74,6 @@ dev = [ "isort>=5.12.0", "mypy>=1.5.0", "types-PyYAML", - "jsonschema>=4.18", ] [tool.black] diff --git a/tests/corpus/classification.yaml b/tests/corpus/classification.yaml deleted file mode 100644 index 798cd5c..0000000 --- a/tests/corpus/classification.yaml +++ /dev/null @@ -1,166 +0,0 @@ -# Server-backed classification contract. Each ALTER gets a fresh seeded table. -table_ddl: >- - CREATE TABLE {table} ( - id UInt64, x UInt64, y UInt64 DEFAULT x + 1, s String, ts DateTime, - INDEX idx id TYPE minmax GRANULARITY 1, - PROJECTION pr (SELECT id ORDER BY id) - ) ENGINE = MergeTree PARTITION BY toYYYYMM(ts) ORDER BY id - TTL ts + INTERVAL 100 YEAR -seed: >- - INSERT INTO {table} (id, x, s, ts) VALUES - (1, 10, 'one', '2020-01-01 00:00:00'), (2, 20, 'two', '2020-02-01 00:00:00') -statements: - - id: create_table - sql: CREATE TABLE IF NOT EXISTS {table} (id UInt64) ENGINE = Memory - expected: metadata - - id: create_view - sql: CREATE VIEW {db}.sample_view AS SELECT id FROM {table} - expected: metadata - - id: drop_table - sql: DROP TABLE IF EXISTS {table} - expected: metadata - - id: rename_table - sql: RENAME TABLE {table} TO {db}.renamed_table - expected: metadata - - id: create_dictionary - sql: CREATE DICTIONARY {db}.lookup (id UInt64) PRIMARY KEY id LAYOUT(FLAT()) - expected: metadata - - id: drop_view - sql: DROP VIEW IF EXISTS {db}.sample_view - expected: metadata - - id: read_only_select - sql: SELECT id FROM {table} - expected: other - - id: insert_select - sql: INSERT INTO {table} SELECT * FROM {db}.source - expected: other - - id: session_setting - sql: SET max_threads = 1 - expected: other - - id: add_column - sql: ALTER TABLE {table} ADD COLUMN added UInt64 - expected: metadata - - id: add_existing_column - sql: ALTER TABLE {table} ADD COLUMN IF NOT EXISTS x UInt64 - expected: metadata - - id: add_default_column - sql: ALTER TABLE {table} ADD COLUMN added UInt64 DEFAULT x + 1 - expected: metadata - - id: add_array_column - sql: ALTER TABLE {table} ADD COLUMN added Array(Tuple(UInt64, String)) - expected: metadata - - id: default_only - sql: ALTER TABLE {table} MODIFY COLUMN y DEFAULT x + 2 - expected: metadata - - id: same_type_default - sql: ALTER TABLE {table} MODIFY COLUMN y UInt64 DEFAULT x + 2 - expected: metadata - without_schema: mutation - - id: comment_modifier - sql: ALTER TABLE {table} MODIFY COLUMN x COMMENT 'mutation UPDATE DELETE is only text' - expected: metadata - - id: comment_column - sql: ALTER TABLE {table} COMMENT COLUMN x 'description' - expected: metadata - - id: modify_setting - sql: ALTER TABLE {table} MODIFY SETTING parts_to_throw_insert = 3000 - expected: metadata - - id: add_index - sql: ALTER TABLE {table} ADD INDEX another x TYPE minmax GRANULARITY 1 - expected: metadata - - id: add_projection - sql: ALTER TABLE {table} ADD PROJECTION another (SELECT id, x ORDER BY x) - expected: metadata - - id: drop_index - sql: ALTER TABLE {table} DROP INDEX idx - expected: mutation - - id: drop_projection - sql: ALTER TABLE {table} DROP PROJECTION pr - expected: mutation - - id: narrow_column_type - sql: ALTER TABLE {table} MODIFY COLUMN x UInt32 - expected: mutation - - id: nullable_column_type - sql: ALTER TABLE {table} MODIFY COLUMN x Nullable(UInt64) - expected: mutation - - id: low_cardinality_column_type - sql: ALTER TABLE {table} MODIFY COLUMN s LowCardinality(String) - expected: mutation - - id: drop_column - sql: ALTER TABLE {table} DROP COLUMN y - expected: mutation - - id: clear_column - sql: ALTER TABLE {table} CLEAR COLUMN y - expected: mutation - - id: materialize_column - sql: ALTER TABLE {table} MATERIALIZE COLUMN y - expected: mutation - - id: rename_column - sql: ALTER TABLE {table} RENAME COLUMN y TO renamed - expected: mutation - - id: update - sql: ALTER TABLE {table} UPDATE x = x + 1 WHERE id = 1 - expected: mutation - - id: delete - sql: ALTER TABLE {table} DELETE WHERE id = 1 - expected: mutation - - id: materialize_index - sql: ALTER TABLE {table} MATERIALIZE INDEX idx - expected: mutation - - id: clear_index - sql: ALTER TABLE {table} CLEAR INDEX idx - expected: mutation - - id: materialize_projection - sql: ALTER TABLE {table} MATERIALIZE PROJECTION pr - expected: mutation - - id: clear_projection - sql: ALTER TABLE {table} CLEAR PROJECTION pr - expected: mutation - - id: modify_ttl - sql: ALTER TABLE {table} MODIFY TTL ts + INTERVAL 99 YEAR - expected: mutation - - id: modify_ttl_without_materialization - sql: ALTER TABLE {table} MODIFY TTL ts + INTERVAL 99 YEAR SETTINGS materialize_ttl_after_modify = 0 - expected: metadata - - id: materialize_ttl - sql: ALTER TABLE {table} MATERIALIZE TTL - expected: mutation - - id: remove_ttl - sql: ALTER TABLE {table} REMOVE TTL - expected: metadata - - id: append_new_sorting_column - sql: ALTER TABLE {table} ADD COLUMN added UInt64, MODIFY ORDER BY (id, added) - expected: metadata - without_schema: rebuild - - id: append_existing_sorting_column - sql: ALTER TABLE {table} MODIFY ORDER BY (id, x) - expected: rebuild - - id: replace_sorting_key - sql: ALTER TABLE {table} MODIFY ORDER BY x - expected: rebuild - - id: change_primary_key - sql: ALTER TABLE {table} MODIFY PRIMARY KEY x - expected: rebuild - - id: change_partition_key - sql: ALTER TABLE {table} MODIFY PARTITION BY toYYYYMM(ts + INTERVAL 1 MONTH) - expected: rebuild - - id: change_engine - sql: ALTER TABLE {table} MODIFY ENGINE = ReplacingMergeTree - expected: rebuild - - id: unchanged_type - sql: ALTER TABLE {table} MODIFY COLUMN x UInt64 - expected: metadata - without_schema: mutation - - id: mixed_metadata_mutation - sql: ALTER TABLE {table} ADD COLUMN added String, MATERIALIZE COLUMN y - expected: mutation - - id: lightweight_delete - sql: DELETE FROM {table} WHERE id = 1 - expected: mutation - - id: lightweight_update - sql: UPDATE {table} SET x = x + 1 WHERE id = 1 - expected: other - detail: lightweight - - id: update_multiple_assignments - sql: ALTER TABLE {table} UPDATE x = x + 1, s = concat(s, '!') WHERE id = 1 - expected: mutation diff --git a/tests/fixtures/env_v0_4_1.py b/tests/fixtures/env_v0_4_1.py index b6048c2..510449f 100644 --- a/tests/fixtures/env_v0_4_1.py +++ b/tests/fixtures/env_v0_4_1.py @@ -164,18 +164,13 @@ def _on_version_apply(ctx, step, heads, run_args): on_version_apply fires AFTER each migration step completes, so we use it for post-migrate hooks only. """ - revision = ( - step.up_revision - if step.is_upgrade - else (step.down_revisions[0] if step.down_revisions else "unknown") + revision = step.up_revision if step.is_upgrade else ( + step.down_revisions[0] if step.down_revisions else "unknown" ) if hook_registry.post_migrate: run_hooks( - ctx.connection, - hook_registry.post_migrate, - db=db, - phase="post_migrate", - revision=revision, + ctx.connection, hook_registry.post_migrate, + db=db, phase="post_migrate", revision=revision, ) with connectable.connect() as connection: @@ -197,11 +192,8 @@ def _on_version_apply(ctx, step, heads, run_args): # Fire pre-migrate hooks before the migration run if hook_registry.pre_migrate: run_hooks( - connection, - hook_registry.pre_migrate, - db=db, - phase="pre_migrate", - revision="all", + connection, hook_registry.pre_migrate, + db=db, phase="pre_migrate", revision="all", ) context.run_migrations() diff --git a/tests/integration/rebuild_spike_probe.py b/tests/integration/rebuild_spike_probe.py deleted file mode 100644 index ea170f3..0000000 --- a/tests/integration/rebuild_spike_probe.py +++ /dev/null @@ -1,281 +0,0 @@ -"""Explicit reproduction of archived A/D SQL and the unchanged E prototype.""" - -import contextlib -import importlib.util -import io -import json -import os -import subprocess -import sys -import time -from concurrent.futures import ThreadPoolExecutor -from pathlib import Path - -import clickhouse_connect -import pytest - -from ch_migrate.sql import split_statements - -SPIKE = Path(__file__).parents[2] / "docs/design/spikes/2026-10-02-rebuild" -pytestmark = pytest.mark.integration - - -def test_rebuild_spike_a_view_names_survive_exchange(project): - script = (SPIKE / "a_exchange_mv.sql").read_text() - for statement in split_statements(script): - project.client.raw_query(statement.sql) - source = project.client.query("SELECT id FROM a.t ORDER BY id").result_rows - old = project.client.query("SELECT id FROM a.t_new ORDER BY id").result_rows - sink = project.client.query("SELECT id FROM a.sink_src ORDER BY id").result_rows - print("A source/old/sink", source, old, sink) - assert source == sink == [(1,), (2,), (3,), (5,)] - assert old == [(1,), (2,), (3,), (4,), (5,)] - - -@pytest.mark.parametrize( - "script", - [ - "d1_inflight_mv.sh", - "d2_async_and_detect.sh", - "d4_inflight_exchange.sh", - "d5_async_exchange.sh", - ], -) -def test_rebuild_spike_d_original_shell_sql(project, script): - result = _run_archived_shell(project, script) - assert result.returncode == 0, result.stdout + result.stderr - print(script, result.stdout) - if script.startswith("d1"): - assert _counts(project.client, "d") == (20010, 10) - elif script.startswith("d2"): - assert _counts(project.client, "d2") == (6003, 6003) - project.client.command("SYSTEM FLUSH LOGS") - rows = project.client.query( - "SELECT tables FROM system.query_log WHERE query_id = 'd3-cascade' AND type = 'QueryStart'" - ).result_rows - assert rows and "d2.feeder" in rows[0][0] and "d2.t" in rows[0][0] - elif script.startswith("d4"): - assert _counts(project.client, "d4") == (15000, 15000) - assert project.client.command("SELECT uniqExact(id) FROM d4.sink") == 15000 - else: - assert _counts(project.client, "d5") == (0, 0) - assert project.client.command("SELECT count() FROM d5.slow_dst") == 6 - errors = project.client.query( - "SELECT status, rows, exception FROM system.asynchronous_insert_log WHERE database = 'd5' AND table = 't'" - ).result_rows - assert sum(row[0] == "FlushError" for row in errors) == 2 - assert all("741" in row[2] for row in errors if row[0] == "FlushError") - - -@pytest.mark.parametrize("trial", [1, 2]) -def test_rebuild_spike_e_original_prototype(project, clickhouse_server, trial): - # Setup, writers, rebuild and finalization share the prototype's server state. - module, clients, trace = _prototype(project, clickhouse_server) - try: - module.setup() - logs = [project.root / f"writer_{kind}.jsonl" for kind in ("sync", "async", "async0")] - rebuild_log = project.root / "rebuild.jsonl" - with ThreadPoolExecutor(max_workers=3) as workers: - futures = [ - workers.submit(module.writer, kind, first, str(path)) - for kind, first, path in zip( - ("sync", "async", "async0"), (10000000, 50000000, 90000000), logs - ) - ] - try: - _wait_for_writers(logs) - module.rebuild(str(rebuild_log)) - time.sleep(1) - finally: - Path(module.STOP).touch() - for future in futures: - future.result(timeout=90) - # A queued fire-and-forget batch is not a lost row; flush before measuring. - project.client.command(f"SYSTEM FLUSH ASYNC INSERT QUEUE {project.database}.t") - captured = io.StringIO() - with contextlib.redirect_stdout(captured): - module.verify([str(path) for path in [*logs, rebuild_log]]) - report = json.JSONDecoder().raw_decode(captured.getvalue())[0] - print("E trial", trial, captured.getvalue()) - assert report["preload_rows"] == 6000000 - assert report["preload_missing"] == report["preload_duplicated"] == 0 - assert report["missing_acked_ids_by_writer"].get("sync", 0) == 0 - assert report["missing_acked_ids_by_writer"].get("async", 0) == 0 - server_report = _server_duplicate_report(project, trace, logs) - print("E server-time evidence", trial, json.dumps(server_report, sort_keys=True)) - assert server_report["duplicate_ids_outside_window"] == 0 - assert server_report["unattributed_duplicate_ids"] == 0 - assert report["dependent_mv_sink_ids_not_exactly_once"] == 0 - assert ( - project.client.command( - f"SELECT sorting_key FROM system.tables WHERE database = '{project.database}' AND name = 't'" - ) - == "k, ts, id" - ) - finally: - for client in clients: - client.close() - - -class _TracingClient: - """Pass through real requests, adding IDs solely for server-time attribution.""" - - def __init__(self, client, trace, kind): - self.client, self.trace, self.kind = client, trace, kind - - def __getattr__(self, name): - return getattr(self.client, name) - - def command(self, query, **kwargs): - if query.startswith(f"CREATE MATERIALIZED VIEW {self.trace['database']}.t_dual "): - kwargs["settings"] = {**kwargs.get("settings", {}), "query_id": self.trace["create_id"]} - return self.client.command(query, **kwargs) - - def insert(self, table, data, **kwargs): - if self.kind: - key = (self.kind, data[0][0]) - query_id = f"{self.trace['prefix']}-{self.kind}-{data[0][0]}" - self.trace["queries"][key] = query_id - kwargs["settings"] = {**kwargs.get("settings", {}), "query_id": query_id} - return self.client.insert(table, data, **kwargs) - - -def _run_archived_shell(project, script): - environment = os.environ.copy() - environment.update( - {"SPIKE_PYTHON": sys.executable, "SPIKE_CLIENT": str(SPIKE / "reproduce_client.py")} - ) - # Only this exact historical invocation is adapted; no Docker command executes. - wrapper = """docker() { - if [ "$1" != exec ] || [ "$2" != spike-rebuild-ch ] || [ "$3" != clickhouse-client ]; then - echo "Refusing an unexpected archived Docker command" >&2; return 2 - fi - shift 3 - "$SPIKE_PYTHON" "$SPIKE_CLIENT" "$@" - } - export -f docker - bash "$1" - """ - return subprocess.run( - ["bash", "-c", wrapper, "spike", str(SPIKE / script)], - cwd=project.root, - env=environment, - text=True, - capture_output=True, - timeout=120, - ) - - -def _counts(client, database): - return client.query( - f"SELECT (SELECT count() FROM {database}.t), (SELECT count() FROM {database}.t_new)" - ).result_rows[0] - - -def _prototype(project, server): - spec = importlib.util.spec_from_file_location("owned_rebuild_prototype", SPIKE / "e2e.py") - module = importlib.util.module_from_spec(spec) - spec.loader.exec_module(module) - clients = [] - trace = {"database": project.database, "prefix": f"spike-{project.database}", "queries": {}} - trace["create_id"] = trace["prefix"] + "-view" - original_log = module.log - - def record_phase(fh, **values): - if values.get("step") == 3 and values.get("phase") == "end": - trace["snapshot_end"] = project.client.command( - "SELECT toUnixTimestamp64Micro(now64(6))" - ) - original_log(fh, **values) - - module.log = record_phase - - def connect(**settings): - client = clickhouse_connect.get_client( - host=server.host, - port=server.port, - username=server.user, - password=server.password, - secure=server.secure, - settings=settings, - send_receive_timeout=60, - ) - clients.append(client) - kind = None - if "async_insert" in settings: - kind = ( - "sync" - if not settings["async_insert"] - else "async" if settings["wait_for_async_insert"] else "async0" - ) - return _TracingClient(client, trace, kind) - - module.client = connect - module.DB = project.database - module.ENGINE = "MergeTree" - module.STOP = str(project.root / "STOP") - module.THROTTLE = "" - return module, clients, trace - - -def _wait_for_writers(paths): - deadline = time.monotonic() + 30 - while time.monotonic() < deadline: - if all(path.exists() and '"t_ack"' in path.read_text() for path in paths): - return - time.sleep(0.05) - raise TimeoutError("Original writers did not acknowledge their initial batches") - - -def _server_duplicate_report(project, trace, paths): - intervals = _server_intervals(project, trace) - start = intervals[trace["create_id"]][0] - end = trace["snapshot_end"] - duplicates = { - row[0] - for row in project.client.query( - f"SELECT id FROM {project.database}.t GROUP BY id HAVING count() > 1" - ).result_rows - } - attributed, outside = set(), set() - for path in paths: - for line in path.read_text().splitlines(): - batch = json.loads(line) - if "t_ack" not in batch: - continue - ids = duplicates.intersection(range(batch["first"], batch["last"] + 1)) - if not ids: - continue - query_id = trace["queries"][(batch["kind"], batch["first"])] - assert query_id in intervals, f"No server timing for {query_id}" - began, finished = intervals[query_id] - attributed.update(ids) - if finished < start or began > end: - outside.update(ids) - return { - "duplicate_ids_inside_window": len(attributed - outside), - "duplicate_ids_outside_window": len(outside), - "unattributed_duplicate_ids": len(duplicates - attributed), - "server_window_start_us": start, - "server_window_end_us": end, - } - - -def _server_intervals(project, trace): - project.client.command("SYSTEM FLUSH LOGS") - rows = project.client.query( - "SELECT query_id, toUnixTimestamp64Micro(query_start_time_microseconds), " - "toUnixTimestamp64Micro(event_time_microseconds) FROM system.query_log " - "WHERE type = 'QueryFinish' AND startsWith(query_id, {prefix:String})", - parameters={"prefix": trace["prefix"]}, - ).result_rows - intervals = {query_id: (start, end) for query_id, start, end in rows} - flushes = project.client.query( - "SELECT query_id, toUnixTimestamp64Micro(flush_time_microseconds) " - "FROM system.asynchronous_insert_log WHERE database = {db:String} " - "AND table = 't' AND status = 'Ok'", - parameters={"db": project.database}, - ).result_rows - for query_id, flushed in flushes: - intervals[query_id] = (flushed, flushed) - return intervals diff --git a/tests/integration/test_classification.py b/tests/integration/test_classification.py deleted file mode 100644 index 29805eb..0000000 --- a/tests/integration/test_classification.py +++ /dev/null @@ -1,69 +0,0 @@ -"""Every corpus ALTER is classified against actual mutation or rejection evidence.""" - -from pathlib import Path - -import pytest -import yaml -from clickhouse_connect.driver.exceptions import DatabaseError - -from ch_migrate.classify import classify -from ch_migrate.introspect import get_live_schema - -CORPUS = yaml.safe_load((Path(__file__).parents[1] / "corpus/classification.yaml").read_text()) -ALTERS = [case for case in CORPUS["statements"] if case["sql"].startswith("ALTER")] -pytestmark = pytest.mark.integration - - -@pytest.mark.parametrize("case", ALTERS, ids=lambda case: case["id"]) -def test_classification_matches_real_alter_work(project, case): - values = {"db": project.database, "table": f"{project.database}.sample"} - project.client.command(CORPUS["table_ddl"].format(**values)) - project.client.command(CORPUS["seed"].format(**values)) - statement = case["sql"].format(**values) - result = classify(statement, get_live_schema(project.client, project.database)) - assert result.kind == case["expected"], (statement, result) - before = _mutation_ids(project) - if result.kind == "rebuild": - code = 36 if "sorting" in case["id"] else 62 - with pytest.raises(DatabaseError, match=f"code: {code},"): - project.client.command(statement) - assert _mutation_ids(project) == before - return - project.client.command(statement, settings={"mutations_sync": 0, "alter_sync": 0}) - created_mutations = _mutation_ids(project) - before - assert bool(created_mutations) == (result.kind == "mutation"), ( - statement, - result, - created_mutations, - ) - - -def test_classification_lightweight_update_is_synchronous_patch_work(project): - table = f"{project.database}.patches" - project.client.command( - f"CREATE TABLE {table} (id UInt64, x UInt64) ENGINE = MergeTree ORDER BY id " - "SETTINGS enable_block_number_column = 1, enable_block_offset_column = 1" - ) - project.client.command(f"INSERT INTO {table} VALUES (1, 10), (2, 20)") - statement = f"UPDATE {table} SET x = x + 1 WHERE id = 1" - result = classify(statement) - project.client.command(statement, settings={"allow_experimental_lightweight_update": 1}) - assert project.client.query(f"SELECT id, x FROM {table} ORDER BY id").result_rows == [ - (1, 11), - (2, 20), - ] - rows = project.client.query( - "SELECT mutation_id FROM system.mutations WHERE database = {db:String} AND table = 'patches'", - parameters={"db": project.database}, - ).result_rows - assert result.kind == "other" and "lightweight" in result.detail - assert rows == [] - - -def _mutation_ids(project): - rows = project.client.query( - "SELECT DISTINCT mutation_id FROM system.mutations " - "WHERE database = {db:String} AND table = {table:String} AND " + project.REWRITES_PARTS, - parameters={"db": project.database, "table": "sample"}, - ).result_rows - return {row[0] for row in rows} diff --git a/tests/integration/test_cloud.py b/tests/integration/test_cloud.py index ab712a7..15dcae5 100644 --- a/tests/integration/test_cloud.py +++ b/tests/integration/test_cloud.py @@ -70,7 +70,7 @@ def test_cloud_version_table_is_shared_and_round_trips(project): def test_cloud_up_advances_version_only_after_mutation_finishes(project): """Cloud cannot hold a mutation with STOP MERGES; a deliberately slow one shows the wait.""" db = project.database - _slow_increment(project, seconds=2) + _slow_update(project, "x + 1", seconds=2) result = project.run("up", "it") assert result.exit_code == 0, result.output assert ( @@ -85,56 +85,63 @@ def test_cloud_up_advances_version_only_after_mutation_finishes(project): assert versioned >= mutated -def test_cloud_killed_up_reattaches_to_its_running_mutation(project): - """SIGKILL mid-wait; the rerun must resume from its journal, never resubmit the UPDATE.""" +def test_cloud_killed_up_waits_for_its_mutation_then_runs_again(project): + """SIGKILL mid-wait; the waived UPDATE's rerun stops until named, then waits and resends.""" db = project.database - _slow_increment(project, seconds=3) - log_path = project.root / "first-up.log" - with log_path.open("w") as log: - first = subprocess.Popen( - [sys.executable, "-m", "ch_migrate.cli", "up", "it"], - cwd=project.root, - stdout=log, - stderr=subprocess.STDOUT, - ) - try: - _until_running_mutation(project) - finally: - first.kill() - first.wait(timeout=10) - resumed = project.run("up", "it") + _slow_update(project, "1", seconds=3) + _kill_up_once_its_mutation_runs(project) + stopped = project.run("up", "it") + assert stopped.exit_code == 1, stopped.output + assert "was interrupted on an earlier run" in stopped.output + resumed = project.run("up", "it", "--rerun-interrupted", "aaaa") assert resumed.exit_code == 0, resumed.output - assert "not reissuing SQL" in resumed.output, resumed.output - increments = project.client.command( + value = project.client.command( f"SELECT x FROM {db}.counter", settings={"select_sequential_consistency": 1} ) - rewrites = project.client.command( - "SELECT uniqExact(mutation_id) FROM system.mutations " - "WHERE database = {db:String} AND table = {table:String} AND " + project.REWRITES_PARTS, + rewrites = project.client.query( + "SELECT create_time, is_done FROM system.mutations " + "WHERE database = {db:String} AND table = {table:String} AND " + + project.REWRITES_PARTS + + " ORDER BY create_time", parameters={"db": db, "table": "counter"}, - ) - print(f"\nCLOUD_SIGKILL increments={increments} owned_mutations={rewrites}") - assert (increments, rewrites) == (1, 1) + ).result_rows + print(f"\nCLOUD_SIGKILL value={value} mutations={len(rewrites)}") + assert value == 1 and len(rewrites) == 2 and all(done for _, done in rewrites) assert project.client.query(f"SELECT version_num FROM {db}.alembic_version").result_rows == [ ("aaaa",) ] -def _slow_increment(project, seconds): +def _slow_update(project, value, seconds): db = project.database project.client.command( f"CREATE TABLE {db}.counter (id UInt64, x UInt64) ENGINE = MergeTree ORDER BY id" ) project.client.command(f"INSERT INTO {db}.counter VALUES (1, 0)") (project.sql_dir / "slow.sql").write_text( - "-- ch-migrate: allow-non-idempotent One controlled increment\n" - f"ALTER TABLE {{db}}.counter UPDATE x = x + 1 + sleepEachRow({seconds}) WHERE id = 1;\n" + "-- ch-migrate: allow-non-idempotent One controlled, slow update\n" + f"ALTER TABLE {{db}}.counter UPDATE x = {value} + sleepEachRow({seconds}) WHERE id = 1;\n" ) project.write_revision( "aaaa", {"upgrade": "from ch_migrate import run_sql\nrun_sql('slow.sql')"} ) +def _kill_up_once_its_mutation_runs(project): + with (project.root / "first-up.log").open("w") as log: + first = subprocess.Popen( + [sys.executable, "-m", "ch_migrate.cli", "up", "it"], + cwd=project.root, + stdout=log, + stderr=subprocess.STDOUT, + ) + try: + _until_running_mutation(project) + finally: + first.kill() + first.wait(timeout=10) + + def _until_running_mutation(project): deadline = time.monotonic() + 60 while time.monotonic() < deadline: diff --git a/tests/integration/test_continuous_insert.py b/tests/integration/test_continuous_insert.py deleted file mode 100644 index 1e17331..0000000 --- a/tests/integration/test_continuous_insert.py +++ /dev/null @@ -1,373 +0,0 @@ -"""Acknowledgement, duplicate-window and loss accounting on owned servers and ClickHouse Cloud.""" - -from __future__ import annotations - -import json -import threading -from contextlib import contextmanager -from dataclasses import dataclass, field, replace -from uuid import uuid4 - -import pytest -import yaml -from test_rebuild_commands import ( - _active_copy, - _baseline, - _copy_after_moved_partition, - _heads, - _helpers, - _hold_command, - _lock_age, - _lock_owner, - _locks, - _operator_release, - _rebuild_result, - _revision, - _running, - _slow_projection, - _target, - _until, - _uuid, -) -from test_rebuild_dependents import _server_intervals - -pytestmark = pytest.mark.integration -_START_ID = 3000000 -_BATCH_SIZE = 18 - - -@pytest.mark.parametrize("mode", ["sync", "kill", "async1", "async0"]) -def test_continuous_insert_single_node(project, clickhouse_server, mode): - _prepare(project, mode) - writer = _exercise(project, clickhouse_server, mode) - _assert_accounting(project, writer) - - -def test_continuous_insert_replicated(cluster_project, clickhouse_cluster): - project, cluster = cluster_project, clickhouse_cluster - engine = f"ReplicatedMergeTree('/chm-continuous/{project.database}', '{{replica}}')" - project.client.command( - f"CREATE TABLE {project.database}.events ON CLUSTER {cluster.name} " - "(id UInt64, ts DateTime, k UInt8, payload String) " - f"ENGINE = {engine} PARTITION BY toYYYYMM(ts) ORDER BY id" - ) - _prepare(project, "replicated", engine) - for client in cluster.clients.values(): - client.command(f"SYSTEM SYNC REPLICA {project.database}.events") - client.command("SYSTEM FLUSH LOGS") - writer = _exercise(project, cluster.servers[1], "replicated") - reports = [] - for client in cluster.clients.values(): - client.command(f"SYSTEM SYNC REPLICA {project.database}.events") - reports.append(_assert_accounting(replace(project, client=client), writer)) - assert reports[0] == reports[1] - assert len(_rebuild_result(project)["new_uuids"]) == 2 - - -@dataclass -class _Writer: - mode: str - stop: threading.Event = field(default_factory=threading.Event) - long_flush: threading.Event = field(default_factory=threading.Event) - gate_query: str | None = None - attempts: dict = field(default_factory=dict) - returned: list = field(default_factory=list) - retries: list = field(default_factory=list) - errors: list = field(default_factory=list) - intervals: dict = field(default_factory=dict) - forced_ids: set = field(default_factory=set) - seed_rows: int = 6000 - - def arm(self): - self.gate_query = None - self.long_flush.set() - - -def _prepare(project, mode, engine="MergeTree()"): - if mode != "replicated": - project.client.command( - f"CREATE TABLE {project.database}.events " - "(id UInt64, ts DateTime, k UInt8, payload String) " - f"ENGINE = {engine} PARTITION BY toYYYYMM(ts) ORDER BY id" - ) - _baseline(project) - project.client.command( - f"INSERT INTO {project.database}.events SELECT number, toDateTime('2026-01-01') + number, " - "number % 7, toString(number) FROM numbers(200)" - ) - rows = 120000 if mode == "kill" else 6000 - project.client.command( - f"INSERT INTO {project.database}.events SELECT 100000 + number, " - "toDateTime('2026-02-01') + number, number % 7, toString(100000 + number) " - f"FROM numbers({rows})" - ) - _revision(project, target=_target(project, engine), select=_slow_projection()) - project.flush_logs() - - -def _exercise(project, server, mode): - writer = _Writer(mode, seed_rows=120000 if mode == "kill" else 6000) - with _writing(server, project, writer): - _until(lambda: len(writer.returned) >= 3, "Writer did not start") - if mode == "async0": - _assert_async_refusal(project, writer) - _revision(project, select=_slow_projection(), allow_async_loss=True) - if mode == "async1": - # Keep a real waiting INSERT visible across both preflight reads. - # Its settings must be read from processes before query_log flushes. - writer.arm() - _until(lambda: _gate_queued(project, writer), "Waiting insert did not queue") - snapshot = f"ALTER TABLE `{project.database}`.`events__chm_snap` ATTACH PARTITION" - prefixes = (snapshot, "EXCHANGE TABLES") if mode.startswith("async") else snapshot - with _hold_command(project, server, prefixes) as gate: - _rebuild_with_writer(project, writer, gate) - after = len(writer.returned) - _until(lambda: len(writer.returned) >= after + 3, "Writer did not continue after swap") - assert not writer.errors, writer.errors - project.drain_async_queue("events") - assert _heads(project) == ["bbbb"] and not _locks(project) - window = _rebuild_result(project)["duplicate_window"] - writer.intervals = _insert_times(project, writer, window) - return writer - - -def _assert_async_refusal(project, writer): - writer.arm() - _until(lambda: _gate_queued(project, writer), "No unacknowledged batch queued") - project.flush_logs() - refused = project.run("up", "it", "--timeout", project.budget(20)) - assert refused.exit_code != 0 and "async" in refused.output.lower(), refused.output - assert not _helpers(project) and not _locks(project) - project.drain_async_queue("events") - - -def _rebuild_with_writer(project, writer, gate): - blocked, release = gate - # Keep process ownership and both real HTTP barriers in one cleanup scope. - with _running(project) as (process, log): - assert blocked.wait(30), log.read_text() - before = len(writer.returned) - _until(lambda: len(writer.returned) >= before + 3, "No batch entered the snapshot window") - writer.forced_ids = { - row_id for query_id in writer.returned[before:] for row_id in writer.attempts[query_id] - } - if writer.mode == "async0": - project.drain_async_queue("events") - blocked.clear() - release.set() - if writer.mode == "kill": - _interrupt_and_resume(project, process) - elif writer.mode.startswith("async"): - _until(lambda: _active_copy(project), "Copy never started") - blocked.clear() - release.clear() - assert blocked.wait(90), log.read_text() - writer.arm() - _until(lambda: _gate_queued(project, writer), "No old-UUID async batch queued") - release.set() - if writer.mode != "kill": - _finish(project, process, log) - - -def _finish(project, process, log): - """Wait for the rebuild; if a swap-race insert stranded rows, act as the operator.""" - if process.wait(timeout=180) == 0: - return - assert "reached only" in log.read_text(), log.read_text() - db = project.database - project.client.command( - f"INSERT INTO {db}.events SELECT * FROM {db}.events__chm_old_bbbb " - f"WHERE id >= {_START_ID} AND id NOT IN (SELECT id FROM {db}.events)" - ) - config = yaml.safe_load((project.root / "config.yaml").read_text()) - cluster = config["environments"]["it"].get("cluster") - scope = f" ON CLUSTER {cluster}" if cluster else "" - project.client.command(f"DROP TABLE {db}._chm_rebuild_lock_events{scope} SYNC") - resumed = project.run("up", "it", "--timeout", project.budget(60)) - assert resumed.exit_code == 0, resumed.output - - -def _interrupt_and_resume(project, process): - _until( - lambda: _copy_after_moved_partition(project, "202601"), - "First partition was not checkpointed before copying the second", - seconds=90, - ) - owner, source_uuid = _lock_owner(project), _uuid(project) - process.kill() - assert process.wait(timeout=15) != 0 - _until(lambda: _lock_age(project) >= 9, "Killed owner's heartbeat did not expire") - refused = project.run("up", "it", "--timeout", project.budget(1)) - assert refused.exit_code != 0 and "reconcil" in refused.output.lower(), refused.output - assert _uuid(project) == source_uuid and _lock_owner(project) == owner - _operator_release(project, process, owner) - with _running(project) as (resumed, log): - _finish(project, resumed, log) - project.flush_logs() - completed = project.client.command( - f"SELECT count() FROM {project.query_log()} WHERE type = 'QueryFinish' AND is_initial_query " - "AND startsWith(query_id, 'chm-rebuild-') AND endsWith(query_id, '202601') " - "AND position(query, {db:String}) > 0", - parameters={"db": project.database}, - ) - assert completed == 1, "Resume recopied the checkpointed partition" - - -@contextmanager -def _writing(server, project, writer): - def send(): - client = server.connect() - try: - batch = 0 - while not writer.stop.is_set(): - _send_batch(client, project, (writer, batch)) - batch += 1 - writer.stop.wait(0.03) - except Exception as exc: - writer.errors.append(exc) - finally: - client.close() - - thread = threading.Thread(target=send, daemon=True) - thread.start() - try: - yield - finally: - writer.stop.set() - thread.join(timeout=30) - assert not thread.is_alive(), "Writer did not stop" - - -def _send_batch(client, project, batch): - writer, number = batch - ids = tuple(range(_START_ID + number * _BATCH_SIZE, _START_ID + (number + 1) * _BATCH_SIZE)) - values = ", ".join( - f"({row_id}, '2026-0{2 + row_id % 3}-02', {row_id % 7}, '{row_id}')" for row_id in ids - ) - for attempt in range(3): - query_id = "chm-continuous-" + uuid4().hex - writer.attempts[query_id] = ids - settings = _writer_settings(writer, query_id, project.cloud) - try: - client.command( - f"INSERT INTO {project.database}.events (id, ts, k, payload) VALUES {values}", - settings=settings, - ) - except Exception as exc: - if writer.mode != "async1" or "Code: 741." not in str(exc) or attempt == 2: - raise - writer.retries.append(query_id) - continue - writer.returned.append(query_id) - return - - -def _writer_settings(writer, query_id, cloud): - delayed = writer.long_flush.is_set() - if delayed: - writer.long_flush.clear() - writer.gate_query = query_id - settings = { - "query_id": query_id, - "log_queries": 1, - "async_insert": int(writer.mode.startswith("async")), - "wait_for_async_insert": int(writer.mode != "async0"), - "async_insert_use_adaptive_busy_timeout": 0, - "async_insert_busy_timeout_ms": 8000 if delayed else 50, - "async_insert_busy_timeout_max_ms": 8000 if delayed else 50, - } - if not cloud: # Cloud locks log_query_settings (already on there). - settings["log_query_settings"] = 1 - return settings - - -def _gate_queued(project, writer): - if writer.gate_query is None: - return False - return project.client.command( - f"SELECT count() FROM {project.everywhere('system.asynchronous_inserts')} " - "WHERE database = {db:String} AND table = 'events' AND has(entries.query_id, {query:String})", - parameters={"db": project.database, "query": writer.gate_query}, - ) - - -def _insert_times(project, writer, window): - if not writer.mode.startswith("async"): - return _server_intervals(project, writer.returned, window) - project.flush_logs() - # As for synchronous inserts, an async insert is in flight from queueing to the end of its - # flush. The logged flush time can follow the snapshot that already holds its rows. - rows = project.client.query( - "SELECT query_id, status, event_time_microseconds <= toDateTime64({end:String}, 6) " - "AND flush_time_microseconds >= toDateTime64({start:String}, 6) " - f"FROM {project.everywhere('system.asynchronous_insert_log')} WHERE database = {{db:String}} " - "AND table = 'events' AND has({ids:Array(String)}, query_id)", - parameters={"db": project.database, "ids": list(writer.attempts), **window}, - ).result_rows - evidence = {query_id: (status, bool(inside)) for query_id, status, inside in rows} - assert len(evidence) == len(rows) and set(evidence) == set(writer.attempts) - assert all(status in ("Ok", "FlushError") for status, _ in evidence.values()) - errors = {query_id for query_id, (status, _) in evidence.items() if status == "FlushError"} - report = _rebuild_result(project)["async_flush_errors"] - assert errors == set(report["query_ids"]) and len(errors) == report["count"] - assert writer.gate_query in errors, "The real old-UUID flush race was not exercised" - if writer.mode == "async1": - assert errors == set(writer.retries), "Every 741 must be retried on the new table" - assert not errors.intersection(writer.returned) - return {query_id: inside for query_id, (status, inside) in evidence.items() if status == "Ok"} - - -def _assert_accounting(project, writer): - rows = project.client.query( - f"SELECT id, count(), groupUniqArray(k), groupUniqArray(payload) " - f"FROM {project.database}.events WHERE id >= {_START_ID} GROUP BY id" - ).result_rows - actual = {row_id for row_id, _, _, _ in rows} - sent = {row_id for query_id in writer.returned for row_id in writer.attempts[query_id]} - assert len(sent) == len(writer.returned) * _BATCH_SIZE - assert actual.issubset(sent) - lost = sent - actual - _assert_loss(project, writer, lost) - owners = { - row_id: query_id for query_id in writer.returned for row_id in writer.attempts[query_id] - } - duplicates = {row_id for row_id, count, _, _ in rows if count > 1} - assert duplicates.intersection(writer.forced_ids), "Snapshot overlap was not exercised" - assert all(writer.intervals[owners[row_id]] for row_id in duplicates) - assert all(count in (1, 2) for _, count, _, _ in rows) - assert all( - keys == [row_id % 7] and payloads == [str(row_id)] for row_id, _, keys, payloads in rows - ) - _assert_seed(project, writer.seed_rows) - assert "ORDER BY (k, ts, id)" in project.client.command( - f"SHOW CREATE TABLE {project.database}.events" - ) - report = { - "mode": writer.mode, - "sent": len(sent), - "duplicates": len(duplicates), - "lost": len(lost), - "retries_741": len(writer.retries), - } - print("CONTINUOUS_INSERT " + json.dumps(report, sort_keys=True), flush=True) - return report - - -def _assert_loss(project, writer, lost): - if writer.mode != "async0": - assert not lost, f"Lost {len(lost)} acknowledged IDs" - return - errors = _rebuild_result(project)["async_flush_errors"] - reported = {row_id for query_id in errors["query_ids"] for row_id in writer.attempts[query_id]} - assert lost == reported and lost, "Every lost row must match an old-UUID FlushError" - assert len(lost) == errors["count"] * _BATCH_SIZE - - -def _assert_seed(project, rows): - actual = project.client.query( - f"SELECT id, count(), any(payload) FROM {project.database}.events " - f"WHERE id < {_START_ID} GROUP BY id" - ).result_rows - expected = set(range(200)) | set(range(100000, 100000 + rows)) - assert {row_id for row_id, _, _ in actual} == expected - assert all(count == 1 and payload == str(row_id) for row_id, count, payload in actual) diff --git a/tests/integration/test_ddl_queue.py b/tests/integration/test_ddl_queue.py deleted file mode 100644 index 6aa40fa..0000000 --- a/tests/integration/test_ddl_queue.py +++ /dev/null @@ -1,332 +0,0 @@ -"""Owned distributed DDL waits, failure propagation, and queue-identity recovery.""" - -import json -import os -import subprocess -import sys -import time -from contextlib import contextmanager - -import pytest - -pytestmark = pytest.mark.integration - - -def test_ddl_queue_create_then_alter_finishes_every_host(cluster_project, clickhouse_cluster): - project, cluster = cluster_project, clickhouse_cluster - _baseline(project) - _revision( - project, - [ - f"CREATE TABLE IF NOT EXISTS {{db}}.owned ON CLUSTER {cluster.name} (id UInt64) ENGINE=Memory", - f"ALTER TABLE {{db}}.owned ON CLUSTER {cluster.name} ADD COLUMN IF NOT EXISTS value String", - ], - ) - result = project.run("up", "it") - assert result.exit_code == 0, result.output - for client in cluster.clients.values(): - assert client.query( - "SELECT name FROM system.columns WHERE database={db:String} AND table='owned' ORDER BY position", - parameters={"db": project.database}, - ).result_rows == [("id",), ("value",)] - rows = _queue(project) - assert len({row[0] for row in rows}) == 2 - assert all(row[3] == "Finished" and row[4] == 0 for row in rows) - assert _heads(project) == ["bbbb"] - - -def test_ddl_queue_down_host_times_out_then_reattaches(cluster_project, clickhouse_cluster): - project, cluster = cluster_project, clickhouse_cluster - _baseline(project) - _revision( - project, - [ - f"CREATE TABLE IF NOT EXISTS {{db}}.owned ON CLUSTER {cluster.name} (id UInt64) ENGINE=Memory" - ], - ) - cluster.stop_node(2) - try: - result = project.run("up", "it", "--timeout", "1") - assert result.exit_code == 1 and "Timed out" in result.output, result.output - assert cluster.hostnames[2] in result.output - before = {row[0] for row in _queue(project)} - assert len(before) == 1 - assert _heads(project) == ["aaaa"] - finally: - cluster.start_node(2) - resumed = project.run("up", "it") - assert resumed.exit_code == 0, resumed.output - assert {row[0] for row in _queue(project)} == before - assert all(row[3] == "Finished" and row[4] == 0 for row in _queue(project)) - assert _heads(project) == ["bbbb"] - - -def test_ddl_queue_host_exception_is_not_finished_success(cluster_project, clickhouse_cluster): - project, cluster = cluster_project, clickhouse_cluster - _baseline(project) - table = f"{project.database}.owned" - project.client.command( - f"CREATE TABLE {table} ON CLUSTER {cluster.name} (id UInt64) ENGINE=Memory" - ) - cluster.clients[2].command(f"ALTER TABLE {table} ADD COLUMN conflict String") - _revision( - project, [f"ALTER TABLE {{db}}.owned ON CLUSTER {cluster.name} ADD COLUMN conflict UInt64"] - ) - result = project.run("up", "it") - assert result.exit_code == 1, result.output - assert cluster.hostnames[2] in result.output and "DUPLICATE_COLUMN" in result.output - assert _heads(project) == ["aaaa"] - assert any(row[3] == "Finished" and row[4] == 15 for row in _queue(project)) - - -def test_ddl_queue_lost_acknowledgement_uses_pre_send_marker(cluster_project, clickhouse_cluster): - project, cluster = cluster_project, clickhouse_cluster - _baseline(project) - _revision( - project, - [ - f"CREATE TABLE IF NOT EXISTS {{db}}.owned ON CLUSTER {cluster.name} (id UInt64) ENGINE=Memory" - ], - ) - code = """import os, signal -from ch_migrate.waiting import MigrationWaiter -original = MigrationWaiter._after -def crash(self, connection, cursor, statement, parameters, context, executemany): - pending = self._pending.get(id(context)) - if pending and pending[1].get('ddl') and pending[1].get('table', [])[-1] == 'owned': - os.kill(os.getpid(), signal.SIGKILL) - return original(self, connection, cursor, statement, parameters, context, executemany) -MigrationWaiter._after = crash -from ch_migrate.cli import main -main() -""" - with _running(project, code) as (process, log): - assert process.wait(timeout=30) == -9, log.read_text() - entries = {row[0] for row in _queue(project)} - assert len(entries) == 1 - resumed = project.run("up", "it") - assert resumed.exit_code == 0, resumed.output - assert {row[0] for row in _queue(project)} == entries - assert _heads(project) == ["bbbb"] - - -def test_ddl_queue_and_replicated_mutation_both_must_finish(cluster_project, clickhouse_cluster): - project, cluster = cluster_project, clickhouse_cluster - _baseline(project) - table = f"{project.database}.owned" - project.client.command( - f"CREATE TABLE {table} ON CLUSTER {cluster.name} (id UInt64, x UInt64) " - f"ENGINE=ReplicatedMergeTree('/clickhouse/ddl_wait/{project.database}', '{{replica}}') ORDER BY id" - ) - project.client.command(f"INSERT INTO {table} VALUES (1, 0)") - cluster.clients[2].command(f"SYSTEM SYNC REPLICA {table}") - _revision( - project, [f"ALTER TABLE {{db}}.owned ON CLUSTER {cluster.name} UPDATE x=x+1 WHERE id=1"] - ) - for client in cluster.clients.values(): - client.command(f"SYSTEM STOP MERGES {table}") - try: - result = project.run("up", "it", "--timeout", "1") - assert result.exit_code == 1, result.output - rows = _queue(project) - assert rows and all(row[3] == "Finished" and row[4] == 0 for row in rows) - assert "parts_to_do=" in result.output - assert _heads(project) == ["aaaa"] - entries = {row[0] for row in rows} - finally: - for client in cluster.clients.values(): - client.command(f"SYSTEM START MERGES {table}") - resumed = project.run("up", "it") - assert resumed.exit_code == 0, resumed.output - assert {row[0] for row in _queue(project)} == entries - for client in cluster.clients.values(): - assert client.command(f"SELECT x FROM {table}") == 1 - assert _heads(project) == ["bbbb"] - - -def test_ddl_queue_missing_ownership_evidence_refuses_resubmission( - cluster_project, clickhouse_cluster -): - project, cluster = cluster_project, clickhouse_cluster - _baseline(project) - _revision( - project, - [ - f"CREATE TABLE IF NOT EXISTS {{db}}.owned ON CLUSTER {cluster.name} (id UInt64) ENGINE=Memory" - ], - ) - cluster.stop_node(2) - try: - assert project.run("up", "it", "--timeout", "1").exit_code == 1 - entries = {row[0] for row in _queue(project)} - row = project.client.query( - f"SELECT generation, position, sequence, payload FROM {project.database}._ch_migrate_journal " - "WHERE revision='bbbb' ORDER BY sequence DESC LIMIT 1" - ).result_rows[0] - payload = json.loads(row[3]) - assert "ddl" in payload - # An operator-restored receipt whose ownership token has no server evidence. - payload["ddl"]["token"] = "missing-ownership-evidence" - project.client.insert( - f"{project.database}._ch_migrate_journal", - [["bbbb", row[0], row[1], row[2] + 1, json.dumps(payload)]], - column_names=["revision", "generation", "position", "sequence", "payload"], - ) - result = project.run("up", "it") - assert result.exit_code == 1 and "unknown" in result.output.lower(), result.output - assert "not reissued" in result.output - assert {row[0] for row in _queue(project)} == entries - assert _heads(project) == ["aaaa"] - finally: - cluster.start_node(2) - - -def test_ddl_queue_is_not_used_for_single_node_migrations(project): - project.write_revision( - "aaaa", - { - "upgrade": 'op.execute(f"CREATE TABLE IF NOT EXISTS {db}.owned (id UInt64) ENGINE=Memory")' - }, - ) - started = project.client.command("SELECT toString(now64(6))") - result = project.run("up", "it") - assert result.exit_code == 0, result.output - project.flush_logs() - # Earlier tests on a shared server (Cloud's ON CLUSTER commands) also read the queue. - reads = project.client.query( - f"SELECT count() FROM {project.query_log()} WHERE type='QueryStart' " - "AND event_time_microseconds >= toDateTime64({started:String}, 6) " - "AND position(query, 'FROM system.distributed_ddl_queue') > 0 " - "AND position(query, 'system.query_log') = 0", - parameters={"started": started}, - ).result_rows[0][0] - assert reads == 0 - - -def test_ddl_queue_ttl_metadata_and_materialization_resume(cluster_project, clickhouse_cluster): - project, cluster = cluster_project, clickhouse_cluster - _baseline(project) - table = f"{project.database}.expiry" - project.client.command( - f"CREATE TABLE {table} ON CLUSTER {cluster.name} (ts DateTime) " - f"ENGINE=ReplicatedMergeTree('/clickhouse/ddl_ttl/{project.database}', '{{replica}}') ORDER BY ts" - ) - project.client.command(f"INSERT INTO {table} VALUES ('2020-01-01 00:00:00')") - cluster.clients[2].command(f"SYSTEM SYNC REPLICA {table}") - _revision( - project, - [f"ALTER TABLE {{db}}.expiry ON CLUSTER {cluster.name} MODIFY TTL ts + INTERVAL 100 YEAR"], - ) - for client in cluster.clients.values(): - client.command(f"SYSTEM STOP MERGES {table}") - try: - result = project.run("up", "it", "--timeout", "1") - assert result.exit_code == 1 and "parts_to_do=" in result.output, result.output - entries = {row[0] for row in _queue(project)} - assert len(entries) == 2 # The metadata and materialization have separate receipts. - assert _heads(project) == ["aaaa"] - finally: - for client in cluster.clients.values(): - client.command(f"SYSTEM START MERGES {table}") - resumed = project.run("up", "it") - assert resumed.exit_code == 0, resumed.output - assert {row[0] for row in _queue(project)} == entries - assert _heads(project) == ["bbbb"] - - -def test_ddl_queue_nonreplicated_mutations_wait_on_each_host(cluster_project, clickhouse_cluster): - project, cluster = cluster_project, clickhouse_cluster - _baseline(project) - table = f"{project.database}.owned" - project.client.command( - f"CREATE TABLE {table} ON CLUSTER {cluster.name} (x UInt64) ENGINE=MergeTree ORDER BY tuple()" - ) - for client in cluster.clients.values(): - client.command(f"INSERT INTO {table} VALUES (0)") - _revision(project, [f"ALTER TABLE {{db}}.owned ON CLUSTER {cluster.name} UPDATE x=x+1 WHERE 1"]) - cluster.clients[2].command(f"SYSTEM STOP MERGES {table}") - try: - result = project.run("up", "it", "--timeout", "1") - assert result.exit_code == 1, result.output - assert cluster.hostnames[2] in result.output - assert _heads(project) == ["aaaa"] - assert cluster.clients[1].command(f"SELECT x FROM {table}") == 1 - assert cluster.clients[2].command(f"SELECT x FROM {table}") == 0 - finally: - cluster.clients[2].command(f"SYSTEM START MERGES {table}") - resumed = project.run("up", "it") - assert resumed.exit_code == 0, resumed.output - for client in cluster.clients.values(): - assert client.command(f"SELECT x FROM {table}") == 1 - - -def test_ddl_queue_cold_version_table_creation_obeys_timeout(cluster_project, clickhouse_cluster): - project, cluster = cluster_project, clickhouse_cluster - project.write_revision("aaaa", {"upgrade": 'op.execute("SELECT 1")'}) - cluster.stop_node(2) - try: - result = project.run("up", "it", "--timeout", "1") - assert result.exit_code == 1 and "Timed out" in result.output, result.output - assert cluster.hostnames[2] in result.output - assert _heads(project) == [] - finally: - cluster.start_node(2) - resumed = project.run("up", "it") - assert resumed.exit_code == 0, resumed.output - assert _heads(project) == ["aaaa"] - - -def _baseline(project): - project.write_revision("aaaa", {"upgrade": 'op.execute("SELECT 1")'}) - result = project.run("up", "it") - assert result.exit_code == 0, result.output - - -def _revision(project, statements): - path = project.sql_dir / "ddl.sql" - path.write_text( - "\n".join( - "-- ch-migrate: allow-non-idempotent Controlled distributed DDL scenario\n" + sql + ";" - for sql in statements - ) - ) - project.write_revision( - "bbbb", {"upgrade": "from ch_migrate import run_sql\nrun_sql('ddl.sql')"}, "aaaa" - ) - - -def _queue(project): - return project.client.query( - "SELECT entry, host, port, status, exception_code FROM system.distributed_ddl_queue " - "WHERE startsWith(settings['log_comment'], 'chm_mutation_') AND position(query, {db:String}) > 0 " - "AND position(query, 'alembic_version') = 0 AND position(query, '_ch_migrate_journal') = 0", - parameters={"db": project.database}, - ).result_rows - - -def _heads(project): - return [ - row[0] - for row in project.client.query( - f"SELECT version_num FROM {project.database}.alembic_version ORDER BY version_num" - ).result_rows - ] - - -@contextmanager -def _running(project, code): - path = project.root / f"ddl-{time.monotonic_ns()}.log" - with path.open("w") as log: - process = subprocess.Popen( - [sys.executable, "-c", code, "up", "it"], - cwd=project.root, - stdout=log, - stderr=subprocess.STDOUT, - env=os.environ.copy(), - ) - try: - yield process, path - finally: - if process.poll() is None: - process.kill() - process.wait(timeout=10) diff --git a/tests/integration/test_drift_workflow.py b/tests/integration/test_drift_workflow.py deleted file mode 100644 index 8e3c41f..0000000 --- a/tests/integration/test_drift_workflow.py +++ /dev/null @@ -1,94 +0,0 @@ -"""Execute the documented workflow's actual shell steps on an owned server.""" - -import json -import os -import shutil -import subprocess -import sys -from pathlib import Path - -import pytest -import yaml -from jsonschema import Draft202012Validator - -pytestmark = pytest.mark.integration -REPO = Path(__file__).parents[2] -EXAMPLE = REPO / "docs/examples/github-actions-drift.yml" - - -@pytest.mark.parametrize("drift", [False, True], ids=["in_sync", "out_of_band_change"]) -def test_drift_workflow_run_steps(project, drift): - project.client.command( - f"CREATE TABLE {project.database}.events (id UInt64) ENGINE = MergeTree ORDER BY id" - ) - captured = project.run("snapshot", "it") - assert captured.exit_code == 0, captured.output - snapshot = next((project.sql_dir / "snapshots").iterdir()) - if drift: - project.client.command( - f"ALTER TABLE {project.database}.events ADD COLUMN unexpected UInt32" - ) - outcomes = _run_steps(project, _workflow_environment(project, snapshot)) - for name in ("install", "credentials", "cleanup"): - assert outcomes[name].returncode == 0, outcomes[name].stderr - assert outcomes["compare"].returncode == int(drift), outcomes["compare"].stderr - assert outcomes["gate"].returncode == int(drift) - document = json.loads((project.root / "drift.json").read_text()) - schema = json.loads((REPO / "docs/schemas/diff.schema.json").read_text()) - Draft202012Validator(schema).validate(document) - assert document["in_sync"] is (not drift) - events = next(item for item in document["objects"] if item["name"] == "events") - assert events["status"] == ("modified" if drift else "in_sync") - assert not (project.root / ".env.local").exists() - - -def test_drift_workflow_passes_actionlint(): - validator = shutil.which("actionlint") - if validator: - command = [validator, str(EXAMPLE)] - elif shutil.which("docker"): - command = ["docker", "run", "--rm", "-i", "rhysd/actionlint:1.7.7", "-"] - else: - pytest.skip("Neither actionlint nor Docker is available to validate the workflow") - result = subprocess.run( - command, input=EXAMPLE.read_text(), capture_output=True, text=True, timeout=120 - ) - assert result.returncode == 0, result.stdout + result.stderr - - -def _workflow_environment(project, snapshot): - environment = os.environ.copy() - password = environment.pop("CH_IT_MIGRATION_PASSWORD") - environment.update( - { - "CH_MIGRATE_PACKAGE": str(REPO), - "CH_MIGRATE_PYTHON": sys.executable, - "CH_MIGRATE_ENV": "it", - "CH_MIGRATE_SNAPSHOT_DIR": str(snapshot), - "DRIFT_MIGRATION_PASSWORD": password, - "UV_TOOL_DIR": str(project.root / "tools"), - "UV_TOOL_BIN_DIR": str(project.root / "bin"), - } - ) - return environment - - -def _run_steps(project, environment): - steps = yaml.safe_load(EXAMPLE.read_text())["jobs"]["drift"]["steps"] - outcomes = {} - for step in steps: - if "run" not in step: - continue - if step["id"] == "gate": - environment["DRIFT_OUTCOME"] = ( - "success" if outcomes["compare"].returncode == 0 else "failure" - ) - outcomes[step["id"]] = subprocess.run( - ["bash", "--noprofile", "--norc", "-e", "-o", "pipefail", "-c", step["run"]], - cwd=project.root, - env=environment, - capture_output=True, - text=True, - timeout=120, - ) - return outcomes diff --git a/tests/integration/test_json_commands.py b/tests/integration/test_json_commands.py deleted file mode 100644 index 7219955..0000000 --- a/tests/integration/test_json_commands.py +++ /dev/null @@ -1,105 +0,0 @@ -"""Versioned documents from actual commands against the owned ClickHouse fixture.""" - -import json -import subprocess -import sys -from pathlib import Path - -import pytest -from jsonschema import Draft202012Validator - -pytestmark = pytest.mark.integration -SCHEMAS = Path(__file__).parents[2] / "docs" / "schemas" - - -def test_json_commands_follow_live_migration_and_drift_state(project): - # One project deliberately follows pending -> applied -> drift transitions. - project.write_revision( - "aaaa", - { - "upgrade": 'op.execute(f"CREATE TABLE IF NOT EXISTS {db}.events (id UInt64) ENGINE = MergeTree ORDER BY id")' - }, - ) - project.write_revision( - "bbbb", - {"upgrade": 'op.execute(f"ALTER TABLE {db}.events ADD COLUMN IF NOT EXISTS extra String")'}, - "aaaa", - ) - result, pending = _document(project, "status") - assert result.exit_code == 1 and pending["pending"] == ["aaaa", "bbbb"] - result, lint = _document(project, "lint") - assert result.exit_code == 0 and lint["counts"]["blocking"] == 0 - upgraded = project.run("up", "it") - assert upgraded.exit_code == 0, upgraded.output - result, status = _document(project, "status") - assert result.exit_code == 0 and status["at_head"] - assert status["current_heads"] == ["bbbb"] - assert status["applied"] == ["aaaa", "bbbb"] - result, history = _document(project, "history") - assert result.exit_code == 0 - assert {item["revision"] for item in history["revisions"] if item["applied"]} == { - "aaaa", - "bbbb", - } - snapshot = project.run("snapshot", "it") - assert snapshot.exit_code == 0, snapshot.output - result, before = _document(project, "diff") - assert result.exit_code == 0 and before["in_sync"], result.output - project.client.command( - f"ALTER TABLE {project.database}.events ADD COLUMN outside_migrations UInt32" - ) - result, after = _document(project, "diff") - assert result.exit_code == 1 and not after["in_sync"] - event = next(item for item in after["objects"] if item["name"] == "events") - assert event["status"] == "modified" - assert any("outside_migrations" in detail["field"] for detail in event["details"]) - _jq_status(project) - - -def test_json_live_lint_exposes_waiver_and_blocking_statement(project): - project.write_revision( - "aaaa", - { - "upgrade": """ - # ch-migrate: allow-non-idempotent Operator reconciles this one-time insert - op.execute(f"INSERT INTO {db}.events SELECT 1") - op.execute(f"CREATE TABLE {db}.events (id UInt64) ENGINE = Memory") - """ - }, - ) - result, document = _document(project, "lint") - assert result.exit_code == 1 - findings = [item for item in document["findings"] if item["rule"] == "idempotency"] - assert any( - item["waived"] == "Operator reconciles this one-time insert" and not item["blocking"] - for item in findings - ) - assert any(item["blocking"] and item["waived"] is None for item in findings) - - -def _document(project, command): - result = project.run(command, "it", "--json") - document = json.loads(result.stdout) - schema = json.loads((SCHEMAS / f"{command}.schema.json").read_text()) - Draft202012Validator(schema).validate(document) - return result, document - - -def _jq_status(project): - result = subprocess.run( - [ - "bash", - "-o", - "pipefail", - "-c", - '"$1" -m ch_migrate.cli status it --json | jq -e .current_heads', - "json-smoke", - sys.executable, - ], - cwd=project.root, - capture_output=True, - text=True, - timeout=30, - ) - assert result.returncode == 0, result.stderr - assert json.loads(result.stdout) == ["bbbb"] diff --git a/tests/integration/test_lint_pending.py b/tests/integration/test_lint_pending.py index d549542..5abdaad 100644 --- a/tests/integration/test_lint_pending.py +++ b/tests/integration/test_lint_pending.py @@ -12,9 +12,7 @@ def test_lint_pending_reports_only_second_revision(project, monkeypatch): (project.sql_dir / "b.sql").write_text( "-- pending change\n\nDROP TABLE IF EXISTS {db}.pending_only;\n" ) - project.write_revision( - "aaaa", {"upgrade": "from ch_migrate import run_sql\nrun_sql('a.sql')"} - ) + project.write_revision("aaaa", {"upgrade": "from ch_migrate import run_sql\nrun_sql('a.sql')"}) project.write_revision( "bbbb", {"upgrade": "from ch_migrate import run_sql\nrun_sql('b.sql')"}, diff --git a/tests/integration/test_mutation_waiting.py b/tests/integration/test_mutation_waiting.py index 81ee5e8..3029298 100644 --- a/tests/integration/test_mutation_waiting.py +++ b/tests/integration/test_mutation_waiting.py @@ -1,288 +1,248 @@ -"""Real-server waiting, crash recovery, failure, and timeout invariants.""" +"""Real-server waiting: `up` records a migration only after the work it started finishes.""" -import json import os import subprocess import sys import time from contextlib import contextmanager -from pathlib import Path import pytest import yaml pytestmark = pytest.mark.integration -CORPUS = yaml.safe_load((Path(__file__).parents[1] / "corpus/classification.yaml").read_text()) -MUTATIONS = [case for case in CORPUS["statements"] if case["expected"] == "mutation"] - - -@pytest.mark.parametrize("case", MUTATIONS, ids=lambda case: case["id"]) -def test_mutation_waits_for_server_corpus(project, case): - values = {"db": project.database, "table": f"{project.database}.sample"} - project.client.command(CORPUS["table_ddl"].format(**values)) - project.client.command(CORPUS["seed"].format(**values)) - if case["id"] == "lightweight_delete": - # Lightweight deletes reject projections under the server's default policy. - project.client.command( - f"ALTER TABLE {values['table']} DROP PROJECTION pr SETTINGS mutations_sync=2" - ) - sql = case["sql"].format(**values) - project.write_revision( - "aaaa", - { - "upgrade": "# ch-migrate: allow-non-idempotent Server-backed mutation corpus\n" - + f"op.execute({sql!r})" - }, - ) - result = project.run("up", "it") - assert result.exit_code == 0, result.output - rows = _mutations(project, "sample") - assert all(row[2] for row in rows), rows - assert any("chm_mutation_" in row[1] for row in rows), rows - assert _heads(project) == ["aaaa"] - -def test_mutation_failure_leaves_head_and_prints_operator_kill(project): - _seed(project) - _upgrade_sql(project, "ALTER TABLE {db}.counter UPDATE x = throwIf(x = 5) WHERE 1") - result = project.run("up", "it") - assert result.exit_code == 1, result.output - rows = _mutations(project) - failed = next(row for row in rows if row[4]) - assert failed[0] in result.output and "throwIf" in result.output - assert "KILL MUTATION WHERE database =" in result.output - assert f"mutation_id = '{failed[0]}'" in result.output - assert _heads(project) == ["aaaa"] - assert any(not row[2] and row[0] == failed[0] for row in _mutations(project)) +# Mutations ClickHouse starts and answers before finishing; STOP MERGES holds each one. +HELD = { + "update": "ALTER TABLE {db}.counter UPDATE x = 1 WHERE id = 1", + "delete": "ALTER TABLE {db}.counter DELETE WHERE id = 5", + "materialize": "ALTER TABLE {db}.counter MATERIALIZE COLUMN doubled", +} -def test_mutation_timeout_preserves_work_and_resumes(project): +@pytest.mark.parametrize("statement", HELD.values(), ids=HELD.keys()) +def test_up_records_a_migration_only_after_its_mutation(project, statement): _seed(project) - _upgrade_sql(project, "ALTER TABLE {db}.counter UPDATE x = x + 1 WHERE id = 1") + _upgrade_sql(project, statement) project.stop_merges(f"{project.database}.counter") try: - result = project.run("up", "it", "--timeout", "0.5") - assert result.exit_code == 1, result.output - rows = _mutations(project) - ids = {row[0] for row in rows} - assert "Timed out" in result.output and "parts_to_do=" in result.output - assert all(mutation in result.output for mutation in ids) + held = project.run("up", "it", "--timeout", "1") + assert held.exit_code == 1 and "Timed out" in held.output, held.output + assert "parts_to_do=" in held.output assert _heads(project) == ["aaaa"] - assert any(not row[2] for row in rows) finally: project.client.command(f"SYSTEM START MERGES {project.database}.counter") - resumed = project.run("up", "it") + resumed = project.run("up", "it", "--rerun-interrupted", "bbbb") assert resumed.exit_code == 0, resumed.output - assert {row[0] for row in _mutations(project)} == ids - assert project.client.command(f"SELECT x FROM {project.database}.counter WHERE id = 1") == 1 + assert all(row[2] for row in _mutations(project)) assert _heads(project) == ["bbbb"] -@pytest.mark.parametrize("trial", range(3)) -def test_mutation_sigkill_rerun_attaches_without_second_increment(project, trial): +def test_interrupted_step_with_a_waiver_stops_until_it_is_named(project): + """A waived statement may already have applied, so a rerun needs someone to check first.""" + db = project.database _seed(project) _upgrade_sql(project, "ALTER TABLE {db}.counter UPDATE x = x + 1 WHERE id = 1") - project.stop_merges(f"{project.database}.counter") + project.stop_merges(f"{db}.counter") try: - with _running(project, ("up", "it")) as (first, first_log): - rows = _until(lambda: _mutations(project), "mutation was not submitted") - mutation = rows[0][0] - _until( - lambda: mutation in first_log.read_text(), "first process did not report waiting" - ) - first.kill() - first.wait(timeout=10) - original = {row[0] for row in _mutations(project)} - assert _heads(project) == ["aaaa"] - with _running(project, ("up", "it")) as (resumed, log): - _until( - lambda: mutation in log.read_text(), "rerun did not report the original mutation" - ) - assert resumed.poll() is None - assert {row[0] for row in _mutations(project)} == original - project.client.command(f"SYSTEM START MERGES {project.database}.counter") - assert resumed.wait(timeout=30) == 0, log.read_text() - assert project.client.command(f"SELECT x FROM {project.database}.counter WHERE id = 1") == 1 - assert _heads(project) == ["bbbb"] + held = project.run("up", "it", "--timeout", "1") + assert held.exit_code == 1 and "Timed out" in held.output, held.output finally: - project.client.command(f"SYSTEM START MERGES {project.database}.counter") + project.client.command(f"SYSTEM START MERGES {db}.counter") + _until(lambda: all(row[2] for row in _mutations(project)), "first increment did not finish") + stopped = project.run("up", "it") + assert stopped.exit_code == 1, stopped.output + assert "was interrupted on an earlier run" in stopped.output + assert "waiver: Controlled waiting acceptance scenario" in stopped.output + assert "--rerun-interrupted bbbb" in stopped.output + assert project.client.command(f"SELECT x FROM {db}.counter WHERE id = 1") == 1 + assert _heads(project) == ["aaaa"] + named = project.run("up", "it", "--rerun-interrupted", "bbbb") + assert named.exit_code == 0, named.output + # Naming the revision accepts the repeat: this increment runs a second time. + assert project.client.command(f"SELECT x FROM {db}.counter WHERE id = 1") == 2 + assert _markers(project) == 0 + assert _heads(project) == ["bbbb"] -def test_mutation_crash_before_durable_receipt_recovers_from_intent(project): +def test_interrupted_step_the_gate_passes_runs_again_on_its_own(project): + db = project.database _seed(project) - _upgrade_sql(project, "ALTER TABLE {db}.counter UPDATE x = x + 1 WHERE id = 1") - # Fault injection kills the real CLI after server acceptance, before journal acknowledgement. - code = """import os, signal -from ch_migrate.waiting import MigrationWaiter -original = MigrationWaiter._after -def crash(self, connection, cursor, statement, parameters, context, executemany): - pending = self._pending.get(id(context)) - if pending and pending[1]['kind'] == 'mutation' and pending[1]['table'][1] == 'counter': - os.kill(os.getpid(), signal.SIGKILL) - return original(self, connection, cursor, statement, parameters, context, executemany) -MigrationWaiter._after = crash -from ch_migrate.cli import main -main() -""" - project.stop_merges(f"{project.database}.counter") + _upgrade_sql(project, "ALTER TABLE {db}.counter MATERIALIZE COLUMN doubled", waived=False) + project.stop_merges(f"{db}.counter") try: - with _running(project, ("up", "it"), code) as (crashed, log): - assert crashed.wait(timeout=30) == -9, log.read_text() - rows = _mutations(project) - original_ids = {row[0] for row in rows} - assert rows and any("chm_mutation_" in row[1] for row in rows) - payload = project.client.query( - f"SELECT argMax(payload, sequence) FROM {project.database}._ch_migrate_journal " - "WHERE revision = 'bbbb' AND position = 1" - ).result_rows[0][0] - assert json.loads(payload)["phase"] == "intent" - project.client.command(f"SYSTEM START MERGES {project.database}.counter") - resumed = project.run("up", "it") - assert resumed.exit_code == 0, resumed.output - assert {row[0] for row in _mutations(project)} == original_ids - assert project.client.command(f"SELECT x FROM {project.database}.counter WHERE id = 1") == 1 + held = project.run("up", "it", "--timeout", "1") + assert held.exit_code == 1 and "Timed out" in held.output, held.output finally: - project.client.command(f"SYSTEM START MERGES {project.database}.counter") + project.client.command(f"SYSTEM START MERGES {db}.counter") + resumed = project.run("up", "it") + assert resumed.exit_code == 0, resumed.output + assert "was interrupted on an earlier run; running it again" in resumed.output + assert _markers(project) == 0 + assert _heads(project) == ["bbbb"] -def test_mutation_foreign_failure_is_not_adopted_or_followed(project): +def test_completed_steps_leave_no_marker(project): _seed(project) - project.client.command( - f"ALTER TABLE {project.database}.counter UPDATE x = throwIf(x = 5) WHERE 1" + _upgrade_sql(project, "ALTER TABLE {db}.counter MATERIALIZE COLUMN doubled", waived=False) + assert project.run("up", "it").exit_code == 0 + assert _markers(project) == 0 + + +def test_direct_alembic_upgrade_runs_the_gate(project): + _seed(project) + _upgrade_sql(project, "ALTER TABLE {db}.counter UPDATE x = x + 1 WHERE id = 1", waived=False) + result = subprocess.run( + [sys.executable, "-m", "alembic", "upgrade", "head"], + cwd=project.root, + env={**os.environ, "CH_ENVIRONMENT": "it"}, + capture_output=True, + text=True, + timeout=120, ) - _until(lambda: any(row[4] for row in _mutations(project)), "foreign mutation did not fail") - original_ids = {row[0] for row in _mutations(project)} - _upgrade_sql(project, "ALTER TABLE {db}.counter UPDATE x = x + 1 WHERE id = 1") - result = project.run("up", "it") - assert result.exit_code == 1 and "foreign mutation" in result.output, result.output - assert {row[0] for row in _mutations(project)} == original_ids + assert result.returncode == 1, result.stdout + result.stderr + assert "Migration gate failed; nothing was applied." in result.stderr + assert "[idempotency]" in result.stdout + result.stderr + assert project.client.command(f"SELECT x FROM {project.database}.counter WHERE id = 1") == 0 assert _heads(project) == ["aaaa"] -def test_mutation_expired_ownership_refuses_instead_of_replaying(project): - _seed( - project, - " SETTINGS finished_mutations_to_keep=1, old_parts_lifetime=0, cleanup_delay_period=1, cleanup_delay_period_random_add=0", +def test_interrupted_downgrade_with_a_waiver_stops_until_it_is_named(project): + db = project.database + _seed(project) + project.write_revision( + "bbbb", + { + "upgrade": 'op.execute("SELECT 1")', + "downgrade": "# ch-migrate: allow-non-idempotent Controlled downgrade backfill\n" + "op.execute(f'ALTER TABLE {db}.counter UPDATE x = x + 1 WHERE id = 5')", + }, + "aaaa", ) - _upgrade_sql(project, "ALTER TABLE {db}.counter UPDATE x = x + 1 WHERE id = 1") - project.stop_merges(f"{project.database}.counter") + assert project.run("up", "it").exit_code == 0 + project.stop_merges(f"{db}.counter") try: - timed_out = project.run("up", "it", "--timeout", "0.4") - assert timed_out.exit_code == 1 + with _running(project, ("down", "it")) as (process, log): + _until(lambda: _mutations(project), "downgrade mutation was not submitted") + process.kill() finally: - project.client.command(f"SYSTEM START MERGES {project.database}.counter") - _until(lambda: all(row[2] for row in _mutations(project)), "owned mutation did not finish") + project.client.command(f"SYSTEM START MERGES {db}.counter") + _until(lambda: all(row[2] for row in _mutations(project)), "downgrade mutation did not finish") + stopped = project.run("down", "it") + assert stopped.exit_code == 1, stopped.output + assert "was interrupted on an earlier run" in stopped.output + assert _heads(project) == ["bbbb"] + named = project.run("down", "it", "--rerun-interrupted", "bbbb") + assert named.exit_code == 0, named.output + assert _heads(project) == ["aaaa"] + + +def test_failed_mutation_is_not_recorded_and_names_its_kill(project): + _seed(project) + _upgrade_sql(project, "ALTER TABLE {db}.counter UPDATE x = throwIf(x = 5) WHERE 1") + result = project.run("up", "it") + assert result.exit_code == 1, result.output + # On Cloud this client may reach a replica that hasn't received the failure reason yet. + failed = _until( + lambda: next((row for row in _mutations(project) if row[4]), None), + "the failed mutation's reason is not visible", + ) + assert failed[0] in result.output and "throwIf" in result.output + assert f"mutation_id = '{failed[0]}'" in result.output + assert _heads(project) == ["aaaa"] + + +def test_killed_mutation_then_fixed_sql_runs_again(project): + """DRY-1604: after the kill, fixing the file and running `up` is the whole recovery.""" + db = project.database project.client.command( - f"ALTER TABLE {project.database}.counter DELETE WHERE 0 SETTINGS mutations_sync=2" + f"CREATE TABLE {db}.orders (id UInt64, raw String, qty UInt64) ENGINE=MergeTree ORDER BY id" ) - _until( - lambda: not any("chm_mutation_" in row[1] for row in _mutations(project)), - "ownership did not expire", + project.client.command(f"INSERT INTO {db}.orders VALUES (1, '3', 0), (2, 'n/a', 0)") + _baseline(project) + path = _upgrade_sql(project, "ALTER TABLE {db}.orders UPDATE qty = toUInt64(raw) WHERE 1") + failed = project.run("up", "it") + assert failed.exit_code == 1 and "Cannot parse" in failed.output, failed.output + blocked = project.run("up", "it") + assert blocked.exit_code == 1, blocked.output + assert "started before this run" in blocked.output and "KILL MUTATION" in blocked.output + mutation = next(row[0] for row in _mutations(project, "orders") if row[4]) + project.client.command( + "KILL MUTATION WHERE database = {db:String} AND mutation_id = {mutation:String}", + parameters={"db": db, "mutation": mutation}, ) - resumed = project.run("up", "it") - assert resumed.exit_code == 1 and "Outcome unknown" in resumed.output, resumed.output - assert "not reissued" in resumed.output - assert project.client.command(f"SELECT x FROM {project.database}.counter WHERE id = 1") == 1 - assert _heads(project) == ["aaaa"] + _until(lambda: not any(row[4] for row in _mutations(project, "orders")), "kill did not land") + _write_sql(path, "ALTER TABLE {db}.orders UPDATE qty = toUInt64OrZero(raw) WHERE 1") + # The killed backfill may have rewritten some parts, so the waived rerun is named. + stopped = project.run("up", "it") + assert stopped.exit_code == 1 and "was interrupted on an earlier run" in stopped.output + fixed = project.run("up", "it", "--rerun-interrupted", "bbbb") + assert fixed.exit_code == 0, fixed.output + rows = project.client.query(f"SELECT id, qty FROM {db}.orders ORDER BY id").result_rows + assert rows == [(1, 3), (2, 0)] + assert _heads(project) == ["bbbb"] -def test_mutation_changed_sql_refuses_replay(project): +def test_rejected_statement_can_be_fixed_and_run_again(project): + """DRY-1604: ClickHouse refuses a sorting-key change; the corrected file then applies.""" _seed(project) - path = _upgrade_sql(project, "ALTER TABLE {db}.counter UPDATE x = x + 1 WHERE id = 1") - project.stop_merges(f"{project.database}.counter") - try: - assert project.run("up", "it", "--timeout", "0.4").exit_code == 1 - before = {row[0] for row in _mutations(project)} - path.write_text(path.read_text().replace("x + 1", "x + 2")) - result = project.run("up", "it") - assert result.exit_code == 1 and "changed" in result.output, result.output - assert {row[0] for row in _mutations(project)} == before - assert _heads(project) == ["aaaa"] - finally: - project.client.command(f"SYSTEM START MERGES {project.database}.counter") + path = _upgrade_sql(project, "ALTER TABLE {db}.counter MODIFY COLUMN id String") + rejected = project.run("up", "it") + assert ( + rejected.exit_code == 1 and "ALTER_OF_COLUMN_IS_FORBIDDEN" in rejected.output + ), rejected.output + assert _heads(project) == ["aaaa"] + _write_sql(path, "ALTER TABLE {db}.counter ADD COLUMN IF NOT EXISTS note String", waived=False) + fixed = project.run("up", "it") + assert fixed.exit_code == 0, fixed.output + assert _heads(project) == ["bbbb"] -def test_mutation_completed_prefix_is_not_repeated_and_downgrade_resets(project): +def test_unfinished_earlier_mutation_holds_the_run(project): + """Work already queued on the database finishes before any migration statement is sent.""" + db = project.database _seed(project) - project.client.command(f"CREATE TABLE {project.database}.audit (n UInt64) ENGINE=Memory") - _upgrade_sql( - project, - "INSERT INTO {db}.audit VALUES (1);\n-- ch-migrate: allow-non-idempotent increment once\nALTER TABLE {db}.counter UPDATE x = x + 1 WHERE id = 1", - ) - project.stop_merges(f"{project.database}.counter") + project.client.command(f"CREATE TABLE {db}.observed (value UInt8) ENGINE=Memory") + _upgrade_sql(project, "INSERT INTO {db}.observed VALUES (1)") + project.stop_merges(f"{db}.counter") try: - assert project.run("up", "it", "--timeout", "0.4").exit_code == 1 + project.client.command(f"ALTER TABLE {db}.counter UPDATE x = 7 WHERE id = 5") + held = project.run("up", "it", "--timeout", "1") + assert held.exit_code == 1 and "Timed out" in held.output, held.output + assert project.client.command(f"SELECT count() FROM {db}.observed") == 0 + assert _heads(project) == ["aaaa"] finally: - project.client.command(f"SYSTEM START MERGES {project.database}.counter") + project.client.command(f"SYSTEM START MERGES {db}.counter") resumed = project.run("up", "it") assert resumed.exit_code == 0, resumed.output - assert project.client.command(f"SELECT count() FROM {project.database}.audit") == 1 - reverted = project.run("down", "it") - assert reverted.exit_code == 0, reverted.output - reapplied = project.run("up", "it") - assert reapplied.exit_code == 0, reapplied.output - assert project.client.command(f"SELECT count() FROM {project.database}.audit") == 2 - assert project.client.command(f"SELECT x FROM {project.database}.counter WHERE id = 1") == 2 + assert project.client.command(f"SELECT count() FROM {db}.observed") == 1 + assert _heads(project) == ["bbbb"] -def test_mutation_waits_for_every_replica(cluster_project, clickhouse_cluster): - project, cluster = cluster_project, clickhouse_cluster - _seed_cluster(project, cluster) - _upgrade_sql(project, "ALTER TABLE {db}.counter UPDATE x = x + 1 WHERE id = 1") - table = f"{project.database}.counter" - for action in ("MERGES", "FETCHES", "REPLICATION QUEUES"): - cluster.clients[2].command(f"SYSTEM STOP {action} {table}") +def test_down_waits_for_its_mutation(project): + db = project.database + _seed(project) + project.write_revision( + "bbbb", + { + "upgrade": 'op.execute("SELECT 1")', + "downgrade": "# ch-migrate: allow-non-idempotent Controlled downgrade backfill\n" + "op.execute(f'ALTER TABLE {db}.counter UPDATE x = 0 WHERE id = 5')", + }, + "aaaa", + ) + assert project.run("up", "it").exit_code == 0 + project.stop_merges(f"{db}.counter") try: - with _running(project, ("up", "it")) as (process, log): - _until(lambda: _mutations(project), "replicated mutation was not submitted") - cluster.clients[2].command(f"SYSTEM SYNC REPLICA {table} PULL") - _until(lambda: all(row[2] for row in _mutations(project)), "initiator did not finish") - second = ( - cluster.clients[2] - .query( - "SELECT is_done FROM system.mutations WHERE database = {db:String} AND table = 'counter'", - parameters={"db": project.database}, - ) - .result_rows - ) - assert second and all(not row[0] for row in second) - _until(lambda: "parts_to_do=" in log.read_text(), "replica wait was not reported") - assert process.poll() is None - for action in ("MERGES", "FETCHES", "REPLICATION QUEUES"): - cluster.clients[2].command(f"SYSTEM START {action} {table}") + with _running(project, ("down", "it")) as (process, log): + _until(lambda: _mutations(project), "downgrade mutation was not submitted") + time.sleep(1) + assert process.poll() is None, log.read_text() + project.client.command(f"SYSTEM START MERGES {db}.counter") assert process.wait(timeout=30) == 0, log.read_text() - for client in cluster.clients.values(): - assert client.command(f"SELECT x FROM {table} WHERE id = 1") == 1 finally: - for action in ("MERGES", "FETCHES", "REPLICATION QUEUES"): - cluster.clients[2].command(f"SYSTEM START {action} {table}") - - -def test_mutation_missing_replica_is_not_success(cluster_project, clickhouse_cluster): - project, cluster = cluster_project, clickhouse_cluster - _seed_cluster(project, cluster) - _upgrade_sql(project, "ALTER TABLE {db}.counter UPDATE x = x + 1 WHERE id = 1") - table = f"{project.database}.counter" - for client in cluster.clients.values(): - client.command(f"SYSTEM STOP MERGES {table}") - try: - first = project.run("up", "it", "--timeout", "0.5") - assert first.exit_code == 1, first.output - original = {row[0] for row in _mutations(project)} - cluster.stop_node(2) - result = project.run("up", "it", "--timeout", "1") - assert result.exit_code == 1 and "Timed out" in result.output, result.output - assert "host" in result.output - assert _heads(project) == ["aaaa"] - assert {row[0] for row in _mutations(project)} == original - finally: - if 2 in cluster.stopped: - cluster.start_node(2) - for client in cluster.clients.values(): - client.command(f"SYSTEM START MERGES {table}") - resumed = project.run("up", "it") - assert resumed.exit_code == 0, resumed.output - assert _heads(project) == ["bbbb"] + project.client.command(f"SYSTEM START MERGES {db}.counter") + assert all(row[2] for row in _mutations(project)) + assert _heads(project) == ["aaaa"] def test_mutation_polling_preserves_session_settings(project): @@ -345,209 +305,28 @@ def test_mutation_raw_bind_paths_are_waited(project, mode): assert _heads(project) == ["bbbb"] -def test_mutation_replaced_table_refuses_resume(project): +def test_mutation_piped_progress_prints_plain_lines(project): + """CI logs and agents get a start line naming the mutation, never a redrawn bar.""" _seed(project) _upgrade_sql(project, "ALTER TABLE {db}.counter UPDATE x=x+1 WHERE id=1") project.stop_merges(f"{project.database}.counter") - result = project.run("up", "it", "--timeout", "0.4") - assert result.exit_code == 1 - project.client.command(f"DROP TABLE {project.database}.counter SYNC") - project.client.command( - f"CREATE TABLE {project.database}.counter (id UInt64, x UInt64) ENGINE=MergeTree ORDER BY id" - ) - project.client.command(f"INSERT INTO {project.database}.counter VALUES (1, 0)") - resumed = project.run("up", "it") - assert resumed.exit_code == 1 and "replaced" in resumed.output, resumed.output - assert project.client.command(f"SELECT x FROM {project.database}.counter") == 0 - assert _heads(project) == ["aaaa"] - - -def test_mutation_healthy_foreign_predecessor_is_waited_before_submission(project): - _seed(project) - _upgrade_sql(project, "ALTER TABLE {db}.counter UPDATE x=x+1 WHERE id=1") - project.stop_merges(f"{project.database}.counter") - project.client.command(f"ALTER TABLE {project.database}.counter UPDATE x=x+10 WHERE id=1") - original = {row[0] for row in _mutations(project)} - try: - blocked = project.run("up", "it", "--timeout", "0.4") - assert blocked.exit_code == 1, blocked.output - assert {row[0] for row in _mutations(project)} == original - assert all("chm_mutation_" not in row[1] for row in _mutations(project)) - assert _heads(project) == ["aaaa"] - finally: - project.client.command(f"SYSTEM START MERGES {project.database}.counter") - resumed = project.run("up", "it") - assert resumed.exit_code == 0, resumed.output - assert project.client.command(f"SELECT x FROM {project.database}.counter WHERE id=1") == 11 - - -@pytest.mark.parametrize("scope", ["session", "query"]) -def test_mutation_ttl_materialization_setting_is_respected(project, scope): - project.client.command( - f"CREATE TABLE {project.database}.expiry (ts DateTime) ENGINE=MergeTree ORDER BY ts" - ) - project.client.command(f"INSERT INTO {project.database}.expiry VALUES ('2020-01-01 00:00:00')") - statement = "ALTER TABLE {db}.expiry MODIFY TTL ts + INTERVAL 100 YEAR" - sql = ( - "SET materialize_ttl_after_modify=0;\n" + statement - if scope == "session" - else statement + " SETTINGS materialize_ttl_after_modify=0" - ) - project.write_revision( - "aaaa", {"upgrade": "from ch_migrate import run_sql\nrun_sql('ttl.sql')"} - ) - (project.sql_dir / "ttl.sql").write_text(sql) - result = project.run("up", "it") - assert result.exit_code == 0, result.output - assert _mutations(project, "expiry") == [] - assert "TTL" in project.client.command(f"SHOW CREATE TABLE {project.database}.expiry") - - -def test_mutation_timeout_budget_is_not_reset_for_next_statement(project): - _seed(project) - project.client.command( - f"CREATE TABLE {project.database}.second (id UInt64, x UInt64) ENGINE=MergeTree ORDER BY id" - ) - project.client.command(f"INSERT INTO {project.database}.second VALUES (1, 0)") - _upgrade_sql( - project, - "ALTER TABLE {db}.counter UPDATE x=x+1 WHERE id=1;\n" - "-- ch-migrate: allow-non-idempotent second controlled update\n" - "ALTER TABLE {db}.second UPDATE x=x+1 WHERE id=1", - ) - project.stop_merges(f"{project.database}.counter") - project.stop_merges(f"{project.database}.second") try: - with _running(project, ("up", "it", "--timeout", "1.5")) as (process, log): - mutation = _until(lambda: _mutations(project), "first statement did not start")[0][0] - _until(lambda: mutation in log.read_text(), "first wait was not visible") - time.sleep(0.7) - project.client.command(f"SYSTEM START MERGES {project.database}.counter") - _until(lambda: _mutations(project, "second"), "second statement did not start") - began_second = time.monotonic() - assert process.wait(timeout=10) == 1, log.read_text() - assert time.monotonic() - began_second < 1.3 - assert "second" in log.read_text() and "Timed out" in log.read_text() - assert _heads(project) == ["aaaa"] + result = subprocess.run( + [sys.executable, "-m", "ch_migrate.cli", "up", "it", "--timeout", "1"], + cwd=project.root, + env={**os.environ, "TERM": "xterm-256color"}, + capture_output=True, + text=True, + timeout=60, + ) + assert result.returncode == 1, result.stderr + start = f" waiting on mutation_2.txt ({project.database}.counter): 0/1 parts" + assert start in result.stderr.splitlines(), result.stderr + assert "\x1b" not in result.stderr and "\r" not in result.stderr + assert "Run ch-migrate up it again: it waits for that work to finish" in result.stderr finally: project.client.command(f"SYSTEM START MERGES {project.database}.counter") - project.client.command(f"SYSTEM START MERGES {project.database}.second") - - -def test_mutation_version_rows_ignore_fire_and_forget_insert_settings(project): - project.write_revision( - "aaaa", - { - "upgrade": 'op.execute("SET async_insert=1, wait_for_async_insert=0, async_insert_use_adaptive_busy_timeout=0, async_insert_busy_timeout_ms=60000")' - }, - ) - project.write_revision("bbbb", {"upgrade": 'op.execute("SELECT 1")'}, "aaaa") - result = project.run("up", "it") - assert result.exit_code == 0, result.output - assert _heads(project) == ["bbbb"] - queued = project.client.query( - "SELECT count() FROM system.asynchronous_inserts WHERE database = {db:String} " - "AND table IN ('alembic_version', '_ch_migrate_journal')", - parameters={"db": project.database}, - ).result_rows[0][0] - assert queued == 0 - - -def test_mutation_version_insert_lost_receipt_finishes_bookkeeping_only(project): - _seed(project) - project.client.command(f"CREATE TABLE {project.database}.audit (n UInt64) ENGINE=Memory") - _upgrade_sql(project, "INSERT INTO {db}.audit VALUES (1)") - code = """import os, signal -from ch_migrate.waiting_versions import VersionWrites -original = VersionWrites._insert -def crash(self, step, checkpoint): - original(self, step, checkpoint) - if step['value'] == 'bbbb': - os.kill(os.getpid(), signal.SIGKILL) -VersionWrites._insert = crash -from ch_migrate.cli import main -main() -""" - with _running(project, ("up", "it"), code) as (process, log): - assert process.wait(timeout=30) == -9, log.read_text() - assert _heads(project) == ["aaaa", "bbbb"] - resumed = project.run("up", "it") - assert resumed.exit_code == 0, resumed.output - assert _heads(project) == ["bbbb"] - audit = project.everywhere(f"{project.database}.audit") - assert project.client.command(f"SELECT count() FROM {audit}") == 1 - - -def test_mutation_transport_loss_is_not_treated_as_server_rejection(project): - _seed(project) - _upgrade_sql(project, "CREATE TABLE IF NOT EXISTS {db}.uncertain (id UInt64) ENGINE=Memory") - code = """from ch_migrate.waiting import MigrationWaiter -from clickhouse_connect.driver.exceptions import OperationalError -original = MigrationWaiter._after -def lose_reply(self, connection, cursor, statement, parameters, context, executemany): - pending = self._pending.get(id(context)) - if pending and pending[1].get('table', [None, None])[-1] == 'uncertain': - raise OperationalError('Connection lost after server accepted the statement') - return original(self, connection, cursor, statement, parameters, context, executemany) -MigrationWaiter._after = lose_reply -from ch_migrate.cli import main -main() -""" - with _running(project, ("up", "it"), code) as (process, log): - assert process.wait(timeout=30) == 1, log.read_text() - assert project.client.command(f"EXISTS TABLE {project.database}.uncertain") == 1 - resumed = project.run("up", "it") - assert resumed.exit_code == 1 and "Outcome unknown" in resumed.output, resumed.output - assert "Read-only inspection:" in resumed.output assert _heads(project) == ["aaaa"] - project.client.command("SYSTEM FLUSH LOGS") - # The client's text continues with the column list; a Shared database also logs its own - # re-execution of the CREATE, under another query_id, with `UUID '...'` inserted there. - prefix = f"CREATE TABLE IF NOT EXISTS {project.database}.uncertain (" - assert _requests_started(project, prefix) == 1 - - -def test_mutation_trailing_foreign_failure_does_not_hold_owned_completion(project): - _seed(project) - _upgrade_sql(project, "ALTER TABLE {db}.counter UPDATE x=x+1 WHERE id=1") - code = """from ch_migrate.waiting import MigrationWaiter -original = MigrationWaiter._after -def append_foreign(self, connection, cursor, statement, parameters, context, executemany): - pending = self._pending.get(id(context)) - if pending and pending[1]['kind'] == 'mutation' and pending[1]['table'][1] == 'counter': - self.client.command(f"ALTER TABLE {self.state.database}.counter UPDATE x=throwIf(x=5) WHERE 1") - return original(self, connection, cursor, statement, parameters, context, executemany) -MigrationWaiter._after = append_foreign -from ch_migrate.cli import main -main() -""" - with _running(project, ("up", "it"), code) as (process, log): - assert process.wait(timeout=30) == 0, log.read_text() - _until(lambda: any(row[4] for row in _mutations(project)), "trailing mutation did not fail") - assert _heads(project) == ["bbbb"] - assert project.client.command(f"SELECT x FROM {project.database}.counter WHERE id=1") == 1 - - -def test_mutation_ttl_resume_does_not_materialize_twice(project): - project.client.command( - f"CREATE TABLE {project.database}.expiry (ts DateTime) ENGINE=MergeTree ORDER BY ts" - ) - project.client.command(f"INSERT INTO {project.database}.expiry VALUES ('2020-01-01 00:00:00')") - project.write_revision("aaaa", {"upgrade": 'op.execute("SELECT 1")'}) - assert project.run("up", "it").exit_code == 0 - _upgrade_sql(project, "ALTER TABLE {db}.expiry MODIFY TTL ts + INTERVAL 100 YEAR") - project.stop_merges(f"{project.database}.expiry") - try: - timed_out = project.run("up", "it", "--timeout", "0.4") - assert timed_out.exit_code == 1, timed_out.output - original = {row[0] for row in _mutations(project, "expiry")} - assert any(not row[2] for row in _mutations(project, "expiry")) - finally: - project.client.command(f"SYSTEM START MERGES {project.database}.expiry") - resumed = project.run("up", "it") - assert resumed.exit_code == 0, resumed.output - assert {row[0] for row in _mutations(project, "expiry")} == original - assert _heads(project) == ["bbbb"] @pytest.mark.skipif(os.name != "posix", reason="A real PTY is required for terminal progress") @@ -564,6 +343,7 @@ def test_mutation_terminal_progress_uses_a_live_line(project): process = subprocess.Popen( [sys.executable, "-m", "ch_migrate.cli", "up", "it", "--timeout", "0.8"], cwd=project.root, + env={**os.environ, "TERM": "xterm-256color"}, stdout=subprocess.DEVNULL, stderr=slave, ) @@ -577,7 +357,9 @@ def test_mutation_terminal_progress_uses_a_live_line(project): chunks.append(os.read(master, 4096)) assert process.wait(timeout=5) == 1 output = b"".join(chunks) - assert b"\x1b[K" in output and b"parts_to_do=" in output + # The live bar redraws in place; the timeout still names what was pending. + assert "━".encode() in output and b" parts" in output + assert b"Timed out" in output and b"parts_to_do=" in output assert _heads(project) == ["aaaa"] finally: if process is not None and process.poll() is None: @@ -589,27 +371,6 @@ def test_mutation_terminal_progress_uses_a_live_line(project): project.client.command(f"SYSTEM START MERGES {project.database}.counter") -def test_mutation_parameterized_ttl_batch_preserves_each_operation(project): - project.client.command( - f"CREATE TABLE {project.database}.expiry (ts DateTime) ENGINE=MergeTree ORDER BY ts" - ) - project.client.command(f"INSERT INTO {project.database}.expiry VALUES ('2020-01-01 00:00:00')") - project.write_revision( - "aaaa", - { - "upgrade": "op.get_bind().exec_driver_sql(" - "f'ALTER TABLE {db}.expiry MODIFY TTL ts + toIntervalYear(%(years)s)', " - "[{'years': 100}, {'years': 99}])" - }, - ) - result = project.run("up", "it") - assert result.exit_code == 0, result.output - rows = _mutations(project, "expiry") - assert all(row[2] for row in rows) - assert len({row[0] for row in rows if "chm_mutation_" in row[1]}) == 2 - assert _heads(project) == ["aaaa"] - - def test_mutation_lightweight_update_finishes_without_fabricating_a_mutation(project): project.client.command( f"CREATE TABLE {project.database}.patches (id UInt64, x UInt64) " @@ -632,6 +393,66 @@ def test_mutation_lightweight_update_finishes_without_fabricating_a_mutation(pro assert _heads(project) == ["aaaa"] +def test_mutation_waits_for_every_replica(cluster_project, clickhouse_cluster): + project, cluster = cluster_project, clickhouse_cluster + _seed_cluster(project, cluster) + _upgrade_sql(project, "ALTER TABLE {db}.counter UPDATE x = x + 1 WHERE id = 1") + table = f"{project.database}.counter" + for action in ("MERGES", "FETCHES", "REPLICATION QUEUES"): + cluster.clients[2].command(f"SYSTEM STOP {action} {table}") + try: + with _running(project, ("up", "it")) as (process, log): + _until(lambda: _mutations(project), "replicated mutation was not submitted") + cluster.clients[2].command(f"SYSTEM SYNC REPLICA {table} PULL") + _until(lambda: all(row[2] for row in _mutations(project)), "initiator did not finish") + second = ( + cluster.clients[2] + .query( + "SELECT is_done FROM system.mutations WHERE database = {db:String} AND table = 'counter'", + parameters={"db": project.database}, + ) + .result_rows + ) + assert second and all(not row[0] for row in second) + _until( + lambda: "waiting on mutation" in log.read_text(), "replica wait was not reported" + ) + assert process.poll() is None + for action in ("MERGES", "FETCHES", "REPLICATION QUEUES"): + cluster.clients[2].command(f"SYSTEM START {action} {table}") + assert process.wait(timeout=30) == 0, log.read_text() + for client in cluster.clients.values(): + assert client.command(f"SELECT x FROM {table} WHERE id = 1") == 1 + finally: + for action in ("MERGES", "FETCHES", "REPLICATION QUEUES"): + cluster.clients[2].command(f"SYSTEM START {action} {table}") + + +def test_mutation_on_a_down_replica_is_not_success(cluster_project, clickhouse_cluster): + project, cluster = cluster_project, clickhouse_cluster + _seed_cluster(project, cluster) + _upgrade_sql(project, "ALTER TABLE {db}.counter UPDATE x = 1 WHERE id = 1") + table = f"{project.database}.counter" + for client in cluster.clients.values(): + client.command(f"SYSTEM STOP MERGES {table}") + try: + first = project.run("up", "it", "--timeout", "0.5") + assert first.exit_code == 1, first.output + cluster.stop_node(2) + result = project.run("up", "it", "--timeout", "1") + assert result.exit_code == 1 and "Timed out" in result.output, result.output + assert "shows 1/2 replicas" in result.output + assert _heads(project) == ["aaaa"] + finally: + if 2 in cluster.stopped: + cluster.start_node(2) + for client in cluster.clients.values(): + client.command(f"SYSTEM START MERGES {table}") + resumed = project.run("up", "it", "--rerun-interrupted", "bbbb") + assert resumed.exit_code == 0, resumed.output + assert _heads(project) == ["bbbb"] + + def _seed_cluster(project, cluster): table = f"{project.database}.counter" cluster.clients[1].command( @@ -645,21 +466,24 @@ def _seed_cluster(project, cluster): assert result.exit_code == 0, result.output -def _seed(project, settings=""): +def _seed(project): project.client.command( - f"CREATE TABLE {project.database}.counter (id UInt64, x UInt64) ENGINE=MergeTree ORDER BY id{settings}" + f"CREATE TABLE {project.database}.counter (id UInt64, x UInt64, " + "doubled UInt64 DEFAULT x * 2) ENGINE=MergeTree ORDER BY id" ) - project.client.command(f"INSERT INTO {project.database}.counter VALUES (1, 0), (5, 5)") + project.client.command(f"INSERT INTO {project.database}.counter (id, x) VALUES (1, 0), (5, 5)") + _baseline(project) + + +def _baseline(project): project.write_revision("aaaa", {"upgrade": 'op.execute("SELECT 1")'}) result = project.run("up", "it") assert result.exit_code == 0, result.output -def _upgrade_sql(project, sql): +def _upgrade_sql(project, sql, waived=True): path = project.sql_dir / "mutation.sql" - path.write_text( - "-- ch-migrate: allow-non-idempotent Controlled waiting acceptance scenario\n" + sql + ";\n" - ) + _write_sql(path, sql, waived) project.write_revision( "bbbb", {"upgrade": "from ch_migrate import run_sql\nrun_sql('mutation.sql')"}, @@ -668,6 +492,15 @@ def _upgrade_sql(project, sql): return path +def _write_sql(path, sql, waived=True): + waiver = "-- ch-migrate: allow-non-idempotent Controlled waiting acceptance scenario\n" + path.write_text((waiver if waived else "") + sql + ";\n") + + +def _markers(project): + return project.client.command(f"SELECT count() FROM {project.database}._ch_migrate_in_progress") + + def _mutations(project, table="counter"): return project.client.query( "SELECT mutation_id, command, is_done, parts_to_do, latest_fail_reason FROM system.mutations " @@ -678,21 +511,6 @@ def _mutations(project, table="counter"): ).result_rows -def _requests_started(project, prefix): - """Requests whose logged text starts with `prefix`, counted on every replica.""" - deadline = time.monotonic() + 30 - while True: - count = project.client.query( - f"SELECT uniqExact(query_id) FROM {project.query_log()} WHERE type = 'QueryStart' " - "AND startsWith(query, {prefix:String})", - parameters={"prefix": prefix}, - settings={"skip_unavailable_shards": 1}, - ).result_rows[0][0] - if count or time.monotonic() > deadline: - return count - time.sleep(1) - - def _heads(project): return [ row[0] diff --git a/tests/integration/test_on_cluster.py b/tests/integration/test_on_cluster.py new file mode 100644 index 0000000..ed345bb --- /dev/null +++ b/tests/integration/test_on_cluster.py @@ -0,0 +1,199 @@ +"""ON CLUSTER statements finish on every host before a migration is recorded.""" + +import pytest + +pytestmark = pytest.mark.integration +NOT_FINISHED = "is not finished on 1 of 2 hosts" + + +def test_on_cluster_create_then_alter_finishes_every_host(cluster_project, clickhouse_cluster): + project, cluster = cluster_project, clickhouse_cluster + _baseline(project) + _revision( + project, + [ + f"CREATE TABLE IF NOT EXISTS {{db}}.owned ON CLUSTER {cluster.name} (id UInt64) ENGINE=Memory", + f"ALTER TABLE {{db}}.owned ON CLUSTER {cluster.name} ADD COLUMN IF NOT EXISTS value String", + ], + ) + result = project.run("up", "it") + assert result.exit_code == 0, result.output + for client in cluster.clients.values(): + assert client.query( + "SELECT name FROM system.columns WHERE database={db:String} AND table='owned' ORDER BY position", + parameters={"db": project.database}, + ).result_rows == [("id",), ("value",)] + assert _heads(project) == ["bbbb"] + + +def test_on_cluster_down_host_times_out_then_runs_again(cluster_project, clickhouse_cluster): + project, cluster = cluster_project, clickhouse_cluster + _baseline(project) + _revision( + project, + [ + f"CREATE TABLE IF NOT EXISTS {{db}}.owned ON CLUSTER {cluster.name} (id UInt64) ENGINE=Memory" + ], + waived=False, + ) + cluster.stop_node(2) + try: + result = project.run("up", "it", "--timeout", "2") + assert result.exit_code == 1 and NOT_FINISHED in result.output, result.output + assert _heads(project) == ["aaaa"] + finally: + cluster.start_node(2) + resumed = project.run("up", "it") + assert resumed.exit_code == 0, resumed.output + for client in cluster.clients.values(): + assert client.command(f"EXISTS TABLE {project.database}.owned") == 1 + assert _heads(project) == ["bbbb"] + + +def test_on_cluster_host_exception_fails_the_run(cluster_project, clickhouse_cluster): + project, cluster = cluster_project, clickhouse_cluster + _baseline(project) + table = f"{project.database}.owned" + project.client.command( + f"CREATE TABLE {table} ON CLUSTER {cluster.name} (id UInt64) ENGINE=Memory" + ) + cluster.clients[2].command(f"ALTER TABLE {table} ADD COLUMN conflict String") + _revision( + project, [f"ALTER TABLE {{db}}.owned ON CLUSTER {cluster.name} ADD COLUMN conflict UInt64"] + ) + result = project.run("up", "it") + assert result.exit_code == 1, result.output + assert cluster.hostnames[2] in result.output and "DUPLICATE_COLUMN" in result.output + assert _heads(project) == ["aaaa"] + + +def test_on_cluster_and_replicated_mutation_both_must_finish(cluster_project, clickhouse_cluster): + project, cluster = cluster_project, clickhouse_cluster + _baseline(project) + table = f"{project.database}.owned" + project.client.command( + f"CREATE TABLE {table} ON CLUSTER {cluster.name} (id UInt64, x UInt64) " + f"ENGINE=ReplicatedMergeTree('/clickhouse/ddl_wait/{project.database}', '{{replica}}') ORDER BY id" + ) + project.client.command(f"INSERT INTO {table} VALUES (1, 0)") + cluster.clients[2].command(f"SYSTEM SYNC REPLICA {table}") + _revision( + project, [f"ALTER TABLE {{db}}.owned ON CLUSTER {cluster.name} UPDATE x=1 WHERE id=1"] + ) + for client in cluster.clients.values(): + client.command(f"SYSTEM STOP MERGES {table}") + try: + # Long enough for the queue entry to finish on both hosts on a slow CI runner, so the + # deadline falls in the mutation wait that STOP MERGES holds open. + result = project.run("up", "it", "--timeout", "5") + assert result.exit_code == 1 and "parts_to_do=" in result.output, result.output + assert _heads(project) == ["aaaa"] + finally: + for client in cluster.clients.values(): + client.command(f"SYSTEM START MERGES {table}") + resumed = project.run("up", "it", "--rerun-interrupted", "bbbb") + assert resumed.exit_code == 0, resumed.output + for client in cluster.clients.values(): + assert client.command(f"SELECT x FROM {table}") == 1 + assert _heads(project) == ["bbbb"] + + +def test_on_cluster_ttl_materialization_is_waited(cluster_project, clickhouse_cluster): + """ClickHouse holds MODIFY TTL until it's materialized, so the DDL limit bounds that wait.""" + project, cluster = cluster_project, clickhouse_cluster + _baseline(project) + table = f"{project.database}.expiry" + project.client.command( + f"CREATE TABLE {table} ON CLUSTER {cluster.name} (ts DateTime) " + f"ENGINE=ReplicatedMergeTree('/clickhouse/ddl_ttl/{project.database}', '{{replica}}') ORDER BY ts" + ) + project.client.command(f"INSERT INTO {table} VALUES ('2020-01-01 00:00:00')") + cluster.clients[2].command(f"SYSTEM SYNC REPLICA {table}") + _revision( + project, + [f"ALTER TABLE {{db}}.expiry ON CLUSTER {cluster.name} MODIFY TTL ts + INTERVAL 100 YEAR"], + waived=False, + ) + for client in cluster.clients.values(): + client.command(f"SYSTEM STOP MERGES {table}") + try: + result = project.run("up", "it", "--timeout", "3") + assert result.exit_code == 1 and "is not finished on" in result.output, result.output + assert _heads(project) == ["aaaa"] + finally: + for client in cluster.clients.values(): + client.command(f"SYSTEM START MERGES {table}") + resumed = project.run("up", "it") + assert resumed.exit_code == 0, resumed.output + for client in cluster.clients.values(): + assert not client.command( + "SELECT count() FROM system.mutations WHERE database = {db:String} AND NOT is_done", + parameters={"db": project.database}, + ) + assert _heads(project) == ["bbbb"] + + +def test_on_cluster_nonreplicated_mutations_wait_on_each_host(cluster_project, clickhouse_cluster): + project, cluster = cluster_project, clickhouse_cluster + _baseline(project) + table = f"{project.database}.owned" + project.client.command( + f"CREATE TABLE {table} ON CLUSTER {cluster.name} (x UInt64) ENGINE=MergeTree ORDER BY tuple()" + ) + for client in cluster.clients.values(): + client.command(f"INSERT INTO {table} VALUES (0)") + _revision(project, [f"ALTER TABLE {{db}}.owned ON CLUSTER {cluster.name} UPDATE x=1 WHERE 1"]) + cluster.clients[2].command(f"SYSTEM STOP MERGES {table}") + try: + result = project.run("up", "it", "--timeout", "2") + assert result.exit_code == 1, result.output + assert cluster.hostnames[2] in result.output + assert _heads(project) == ["aaaa"] + assert cluster.clients[1].command(f"SELECT x FROM {table}") == 1 + assert cluster.clients[2].command(f"SELECT x FROM {table}") == 0 + finally: + cluster.clients[2].command(f"SYSTEM START MERGES {table}") + resumed = project.run("up", "it", "--rerun-interrupted", "bbbb") + assert resumed.exit_code == 0, resumed.output + for client in cluster.clients.values(): + assert client.command(f"SELECT x FROM {table}") == 1 + + +def test_on_cluster_cold_version_table_creation_obeys_timeout(cluster_project, clickhouse_cluster): + project, cluster = cluster_project, clickhouse_cluster + project.write_revision("aaaa", {"upgrade": 'op.execute("SELECT 1")'}) + cluster.stop_node(2) + try: + result = project.run("up", "it", "--timeout", "1") + assert result.exit_code == 1 and NOT_FINISHED in result.output, result.output + assert _heads(project) == [] + finally: + cluster.start_node(2) + resumed = project.run("up", "it") + assert resumed.exit_code == 0, resumed.output + assert _heads(project) == ["aaaa"] + + +def _baseline(project): + project.write_revision("aaaa", {"upgrade": 'op.execute("SELECT 1")'}) + result = project.run("up", "it") + assert result.exit_code == 0, result.output + + +def _revision(project, statements, waived=True): + """Waived statements need --rerun-interrupted after an interruption; the rest rerun alone.""" + waiver = "-- ch-migrate: allow-non-idempotent Controlled distributed DDL scenario\n" + path = project.sql_dir / "ddl.sql" + path.write_text("\n".join((waiver if waived else "") + sql + ";" for sql in statements)) + project.write_revision( + "bbbb", {"upgrade": "from ch_migrate import run_sql\nrun_sql('ddl.sql')"}, "aaaa" + ) + + +def _heads(project): + return [ + row[0] + for row in project.client.query( + f"SELECT version_num FROM {project.database}.alembic_version ORDER BY version_num" + ).result_rows + ] diff --git a/tests/integration/test_plan_commands.py b/tests/integration/test_plan_commands.py deleted file mode 100644 index 8d8beeb..0000000 --- a/tests/integration/test_plan_commands.py +++ /dev/null @@ -1,329 +0,0 @@ -"""Plan acceptance against live owned ClickHouse objects and revision state.""" - -import json -import secrets -import time -from pathlib import Path - -import pytest -import yaml -from jsonschema import Draft202012Validator - -pytestmark = pytest.mark.integration -REPO = Path(__file__).parents[2] -CORPUS = yaml.safe_load((REPO / "tests/corpus/classification.yaml").read_text()) - - -def test_plan_classifies_every_pending_corpus_statement(project): - database = project.database - table = f"{database}.sample" - project.client.command(CORPUS["table_ddl"].format(db=database, table=table)) - project.client.command(CORPUS["seed"].format(db=database, table=table)) - project.client.command("SYSTEM FLUSH LOGS") # Seed observable writer history, outside plan. - sql = "\n".join( - case["sql"].format(db=database, table=table) + ";" for case in CORPUS["statements"] - ) - (project.sql_dir / "corpus.sql").write_text(sql + "\n") - project.write_revision( - "corpus", - {"upgrade": "from ch_migrate import run_sql\nrun_sql('corpus.sql')"}, - ) - result, document = _plan(project) - assert result.exit_code == 1, result.output # Unwaived corpus actions block up. - assert document["gate_would_refuse"] is True - statements = document["migrations"][0]["statements"] - assert [item["sql"].strip().rstrip(";") for item in statements] == [ - case["sql"].format(db=database, table=table) for case in CORPUS["statements"] - ] - for case, statement in zip(CORPUS["statements"], statements): - assert statement["classification"]["kind"] == case["expected"], case["id"] - assert statement["line"] >= 1 and statement["file"].endswith("corpus.sql") - if "detail" in case: - assert case["detail"] in statement["classification"]["detail"] - assert project.client.command(f"EXISTS TABLE {database}.alembic_version") == 0 - - -def test_plan_orders_pending_branches_after_applied_ancestors(project): - _revision( - project, "root", None, "CREATE TABLE IF NOT EXISTS {db}.root (id UInt64) ENGINE = Memory" - ) - _revision( - project, "left", "root", "CREATE TABLE IF NOT EXISTS {db}.left (id UInt64) ENGINE = Memory" - ) - _revision( - project, - "right", - "root", - "CREATE TABLE IF NOT EXISTS {db}.right (id UInt64) ENGINE = Memory", - ) - _revision( - project, - "merged", - ("left", "right"), - "CREATE TABLE IF NOT EXISTS {db}.merged (id UInt64) ENGINE = Memory", - ) - applied = project.run("up", "it", "-r", "right") - assert applied.exit_code == 0, applied.output - result, document = _plan(project) - assert result.exit_code == 0, result.output - assert [item["revision"] for item in document["migrations"]] == ["left", "merged"] - assert all(len(item["statements"]) == 1 for item in document["migrations"]) - assert all( - item["statements"][0]["sql"].startswith("CREATE TABLE") for item in document["migrations"] - ) - assert document["gate_would_refuse"] is False - assert project.client.command(f"EXISTS TABLE {project.database}.left") == 0 - - -def test_plan_reports_exact_column_bytes_and_table_and_partition_ceilings(project): - table = f"{project.database}.events" - project.client.command( - f"CREATE TABLE {table} (id UInt64, x UInt64, payload String, ts Date) " - "ENGINE = MergeTree PARTITION BY toYYYYMM(ts) ORDER BY id" - " SETTINGS min_rows_for_wide_part = 0, min_bytes_for_wide_part = 0" - ) - for month in (1, 2): - project.client.command( - f"INSERT INTO {table} VALUES ({month}, {month}, " - f"'{('row' * (month * 200))}', '2020-{month:02d}-01')" - ) - sql = ( - f"ALTER TABLE {table} MODIFY COLUMN x UInt32;\n" - f"ALTER TABLE {table} DROP COLUMN payload;\n" - f"ALTER TABLE {table} UPDATE x = x + 1 WHERE id = 1;\n" - f"ALTER TABLE {table} UPDATE x = x + 1 IN PARTITION ID '202001' WHERE id = 1;\n" - ) - (project.sql_dir / "sizes.sql").write_text(sql) - project.write_revision( - "sizes", {"upgrade": "from ch_migrate import run_sql\nrun_sql('sizes.sql')"} - ) - result, document = _plan(project) - assert result.exit_code == 1, result.output - sizes = [item["size"] for item in document["migrations"][0]["statements"]] - assert sizes[0]["precision"] == sizes[1]["precision"] == "exact" - assert sizes[0]["columns"] == ["x"] - assert sizes[1]["columns"] == ["payload"] - _assert_size(sizes[0], _column_stats(project, "events", "x")) - _assert_size(sizes[1], _column_stats(project, "events", "payload")) - assert sizes[2]["precision"] == sizes[3]["precision"] == "ceiling" - assert sizes[2]["partition_id"] is None - assert sizes[3]["partition_id"] == "202001" - _assert_size(sizes[2], _column_stats(project, "events")) - _assert_size(sizes[3], _column_stats(project, "events", partition="202001")) - human = project.run("plan", "it") - assert human.exit_code == result.exit_code - for value in ("events", "sizes.sql", "exact", "ceiling", "x", "payload"): - assert value.lower() in human.output.lower(), human.output - assert "up to" in human.output - for size in sizes: - for field in ("compressed_bytes", "uncompressed_bytes", "part_count"): - assert str(size[field]) in human.output - assert "202001" in human.output - - -def test_plan_compact_column_bytes_are_a_ceiling_not_zero_work(project): - table = f"{project.database}.compact" - project.client.command( - f"CREATE TABLE {table} (id UInt64, x UInt64) ENGINE = MergeTree ORDER BY id " - "SETTINGS min_rows_for_wide_part = 1000000, min_bytes_for_wide_part = 100000000" - ) - project.client.command(f"INSERT INTO {table} VALUES (1, 2)") - _revision(project, "compact", None, f"ALTER TABLE {table} MODIFY COLUMN x UInt32") - result, document = _plan(project) - assert result.exit_code == 0, result.output - size = document["migrations"][0]["statements"][0]["size"] - assert size["precision"] == "ceiling" - assert size["columns"] == ["x"] - _assert_size(size, _column_stats(project, "compact")) - - -@pytest.mark.parametrize( - ("predicate", "partition"), - [ - ("toYYYYMM(ts) = 202001", "202001"), - ("_partition_id = '202001'", "202001"), - ("toYYYYMM(ts) = 202001 OR id = 2", None), - ], -) -def test_plan_narrows_only_proven_partition_predicates(project, predicate, partition): - table = f"{project.database}.events" - project.client.command( - f"CREATE TABLE {table} (id UInt64, ts Date) ENGINE = MergeTree PARTITION BY toYYYYMM(ts) ORDER BY id" - ) - project.client.command(f"INSERT INTO {table} VALUES (1, '2020-01-01'), (2, '2020-02-01')") - _revision(project, "partition", None, f"DELETE FROM {table} WHERE {predicate}") - result, document = _plan(project) - assert result.exit_code == 1, result.output - size = document["migrations"][0]["statements"][0]["size"] - assert size["precision"] == "ceiling" - assert size["partition_id"] == partition - _assert_size(size, _column_stats(project, "events", partition=partition)) - - -def test_plan_reports_actual_mv_and_dictionary_dependents(project, request): - database = project.database - source = f"{database}.source" - project.client.command( - f"CREATE TABLE {source} (id UInt64, value UInt64) ENGINE = MergeTree ORDER BY id" - ) - project.client.command( - f"CREATE TABLE {database}.sink (id UInt64) ENGINE = MergeTree ORDER BY id" - ) - project.client.command( - f"CREATE MATERIALIZED VIEW {database}.source_mv TO {database}.sink " - f"AS SELECT id FROM {source}" - ) - dictionary_user = _dictionary_source_user(project, request) - project.client.command( - f"CREATE DICTIONARY {database}.source_dict (id UInt64, value UInt64) " - "PRIMARY KEY id " - f"SOURCE(CLICKHOUSE(DB '{database}' TABLE 'source' {dictionary_user})) " - "LIFETIME(MIN 0 MAX 0) LAYOUT(FLAT())" - ) - _revision(project, "deps", None, f"ALTER TABLE {source} DROP COLUMN IF EXISTS value") - result, document = _plan(project) - assert result.exit_code == 0, result.output - statement = document["migrations"][0]["statements"][0] - dependencies = {(item["name"].split(".")[-1], item["type"]) for item in statement["downstream"]} - assert ("source_mv", "materialized_view") in dependencies - assert ("source_dict", "dictionary") in dependencies - human = project.run("plan", "it") - assert human.exit_code == 0, human.output - for name in ("source_mv", "source_dict", "DROP COLUMN", "deps"): - assert name in human.output - - -def test_plan_gate_only_blocks_gate_rules_and_reports_all_findings(project): - table = f"{project.database}.events" - project.client.command(f"CREATE TABLE {table} (id UInt64) ENGINE = MergeTree ORDER BY id") - path = project.sql_dir / "gate.sql" - path.write_text( - f"DROP TABLE IF EXISTS {table};\n" - f"CREATE TABLE {table} (id UInt64) ENGINE = MergeTree ORDER BY id;\n" - ) - project.write_revision( - "gate", {"upgrade": "from ch_migrate import run_sql\nrun_sql('gate.sql')"} - ) - result, document = _plan(project) - assert result.exit_code == 1, result.output - assert document["gate_would_refuse"] is True - assert document["counts"]["blocking"] >= 1 - findings = document["findings"] - assert {item["rule"] for item in findings} >= {"idempotency", "destructive_changes"} - assert any(item["rule"] == "idempotency" and item["blocking"] for item in findings) - assert all(not item["blocking"] for item in findings if item["rule"] == "destructive_changes") - assert any(item["file"].endswith("gate.sql") and item["line"] == 2 for item in findings) - assert any( - item["rule"] == "idempotency" - for item in document["migrations"][0]["statements"][1]["findings"] - ) - human = project.run("plan", "it") - assert human.exit_code == 1 - assert "idempotency" in human.output and "destructive_changes" in human.output - - -def test_plan_under_read_only_user_emits_only_inspection_queries(project, monkeypatch): - database = project.database - table = f"{database}.events" - project.client.command( - f"CREATE TABLE {table} (id UInt64, x UInt64) ENGINE = MergeTree ORDER BY id" - ) - project.client.command(f"INSERT INTO {table} VALUES (1, 5)") - _revision(project, "inspect", None, f"ALTER TABLE {table} MODIFY COLUMN x UInt32") - user = "plan_audit_" + secrets.token_hex(6) - password = project.new_password() - project.client.command(f"CREATE USER {user} IDENTIFIED WITH sha256_password BY '{password}'") - try: - # Cloud's admin may grant *.* only within its own grants (CURRENT GRANTS). - project.client.command( - f"GRANT CURRENT GRANTS(SELECT, SHOW TABLES, SHOW DATABASES ON *.*) TO {user}" - ) - config_path = project.root / "config.yaml" - config = yaml.safe_load(config_path.read_text()) - config["environments"]["it"]["migration_user"] = user - config_path.write_text(yaml.safe_dump(config)) - monkeypatch.setenv("CH_IT_MIGRATION_PASSWORD", password) - result, document = _plan(project) - assert result.exit_code == 0, result.output - assert document["migrations"][0]["statements"][0]["classification"]["kind"] == "mutation" - assert project.client.command(f"EXISTS TABLE {database}.alembic_version") == 0 - project.client.command("SYSTEM FLUSH LOGS") # Audit observer; not the plan user. - rows = _plan_user_query_kinds(project, user) - assert rows, "Plan user has no QueryFinish entries; read-only audit is inconclusive" - assert {row[0] for row in rows} <= {"Select", "Show", "Describe", "Explain"}, rows - finally: - project.client.command(f"DROP USER IF EXISTS {user}") - - -def _plan(project): - result = project.run("plan", "it", "--json") - document = json.loads(result.stdout) - schema = json.loads((REPO / "docs/schemas/plan.schema.json").read_text()) - Draft202012Validator.check_schema(schema) - Draft202012Validator(schema).validate(document) - assert document["command"] == "plan" and document["schema_version"] == 1 - if "error" not in document: - assert document["database"] == project.database - assert document["environment"] == "it" - return result, document - - -def _revision(project, revision, parent, sql): - sql = sql.replace("{db}", project.database) - project.write_revision( - revision, {"upgrade": f"from alembic import op\nop.execute({sql!r})"}, parent - ) - - -def _dictionary_source_user(project, request): - """Cloud requires a non-default dictionary source user; use a throwaway SELECT-only one.""" - if not project.cloud: - return "" - user, password = project.database + "_dict", project.new_password() - request.addfinalizer(lambda: project.client.command(f"DROP USER IF EXISTS {user}")) - project.client.command(f"CREATE USER {user} IDENTIFIED WITH sha256_password BY '{password}'") - project.client.command(f"GRANT SELECT ON {project.database}.source TO {user}") - return f"USER '{user}' PASSWORD '{password}'" - - -def _plan_user_query_kinds(project, user): - # On Cloud the plan's queries may land on another replica, whose log flushes on its own timer. - deadline = time.monotonic() + 30 - while True: - rows = project.client.query( - f"SELECT DISTINCT query_kind FROM {project.query_log()} " - "WHERE type = 'QueryFinish' AND user = {audit_user:String}", - parameters={"audit_user": user}, - settings={"skip_unavailable_shards": 1}, - ).result_rows - if rows or time.monotonic() > deadline: - return rows - time.sleep(1) - - -def _column_stats(project, table, column=None, partition=None): - clauses = ["database = {db:String}", "table = {table:String}", "active = 1"] - params = {"db": project.database, "table": table} - if column is not None: - clauses.append("column = {column:String}") - params["column"] = column - if partition is not None: - clauses.append("partition_id = {partition:String}") - params["partition"] = partition - columns = ( - "sum(column_data_compressed_bytes), sum(column_data_uncompressed_bytes), uniqExact(name)" - ) - source = "system.parts_columns" - if column is None: - columns = "sum(data_compressed_bytes), sum(data_uncompressed_bytes), count()" - source = "system.parts" - rows = project.client.query( - f"SELECT {columns} FROM {source} WHERE " + " AND ".join(clauses), parameters=params - ).result_rows - return tuple(int(value) for value in rows[0]) - - -def _assert_size(actual, expected): - assert actual["compressed_bytes"] == expected[0] - assert actual["uncompressed_bytes"] == expected[1] - assert actual["part_count"] == expected[2] diff --git a/tests/integration/test_rebuild_commands.py b/tests/integration/test_rebuild_commands.py deleted file mode 100644 index 8d9c641..0000000 --- a/tests/integration/test_rebuild_commands.py +++ /dev/null @@ -1,935 +0,0 @@ -"""Rebuild acceptance on the fixtures' owned servers and, where portable, ClickHouse Cloud.""" - -from __future__ import annotations - -import http.client -import json -import os -import secrets -import signal -import ssl -import subprocess -import sys -import threading -import time -from contextlib import contextmanager -from dataclasses import replace -from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer -from urllib.parse import parse_qs, urlsplit -from uuid import uuid4 - -import certifi -import pytest -import yaml - -pytestmark = pytest.mark.integration - - -@pytest.fixture -def owned_project(request): - if os.environ.get("CH_MIGRATE_IT_URL"): - pytest.skip("Rebuild acceptance requires the fixture-owned Docker server") - return request.getfixturevalue("project") - - -def test_rebuild_sorting_key_preserves_rows_and_rollback(project): - before = _seed(project, 12000) - original_uuid = _uuid(project) - _revision(project) - applied = project.run("up", "it") - assert applied.exit_code == 0, applied.output - assert "ORDER BY (k, ts, id)" in project.client.command( - f"SHOW CREATE TABLE {project.database}.events" - ) - assert _checksum(project) == before - helpers = _helpers(project) - assert len(helpers) == 1, helpers - result = _rebuild_result(project) - assert result["rollback_table"] == helpers[0] - assert _checksum(project, helpers[0]) == before - assert original_uuid in result["old_uuids"].values() - assert _uuid(project) in result["new_uuids"].values() - assert _uuid(project) != original_uuid - assert result["duplicate_window"]["start"] <= result["duplicate_window"]["end"] - assert set(result["async_flush_errors"]) == { - "count", - "bytes", - "query_ids", - "complete", - "replicas", - } - assert result["async_flush_errors"]["complete"] is not project.cloud - assert _heads(project) == ["bbbb"] - rerun = project.run("up", "it") - assert rerun.exit_code == 0, rerun.output - assert _checksum(project) == before - assert _helpers(project) == helpers - project.client.command(f"DROP TABLE {project.database}.{result['rollback_table']} SYNC") - assert _helpers(project) == [] - assert _checksum(project) == before - - -def test_rebuild_sigkill_requires_operator_release_then_resumes(project): - before = _seed(project, 120000, first_partition_rows=200) - source_uuid = _uuid(project) - _revision(project, select=_slow_projection()) - first_partition = "202601" - with _running(project) as (first, log): - orphan = _until( - lambda: _copy_after_moved_partition(project, first_partition), - "First partition never moved before the second partition copy", - seconds=90, - ) - owner = _lock_owner(project) - first.kill() - assert first.wait(timeout=15) != 0 - assert first.poll() is not None - assert _until(lambda: _active_copy(project) == orphan, "Server-side orphan disappeared") - _assert_locked_refusal(project, owner, orphan) - _until(lambda: _lock_age(project) >= 9, "Killed owner's heartbeat did not expire", 30) - _assert_locked_refusal(project, owner, orphan) - assert _heads(project) == ["aaaa"] - assert _uuid(project) == source_uuid - _operator_release(project, first, owner) - resumed = project.run("up", "it", "--timeout", "180") - assert resumed.exit_code == 0, resumed.output + log.read_text() - assert not _active_copy(project) - assert not _locks(project) - assert _checksum(project) == before - assert _partition_rows(project, "events", first_partition) == 200 - project.flush_logs() - completed = project.client.command( - f"SELECT count() FROM {project.query_log()} WHERE type = 'QueryFinish' AND is_initial_query " - "AND startsWith(query_id, 'chm-rebuild-') AND endsWith(query_id, {part:String}) " - "AND position(query, {db:String}) > 0", - parameters={"part": first_partition, "db": project.database}, - ) - assert completed == 1, "Resume must not recopy an already moved partition" - - -def test_rebuild_stale_live_owner_cannot_be_supplanted_before_exchange(project): - before = _seed(project, 120000) - original_uuid = _uuid(project) - _revision(project, select=_slow_projection()) - with _running(project) as (first, log): - orphan = _until(lambda: _active_copy(project), "Owner never started copying") - owner = _lock_owner(project) - os.kill(first.pid, signal.SIGSTOP) - try: - _until(lambda: _lock_age(project) >= 9, "Paused owner's heartbeat did not expire", 30) - _assert_locked_refusal(project, owner, orphan) - assert _uuid(project) == original_uuid, "Competitor swapped while owner was paused" - assert first.poll() is None - finally: - os.kill(first.pid, signal.SIGCONT) - assert first.wait(timeout=180) == 0, log.read_text() - assert _checksum(project) == before - assert _uuid(project) != original_uuid - assert not _locks(project) - - -def test_rebuild_concurrent_up_refuses_second_owner(project): - before = _seed(project, 80000) - _revision(project, select=_slow_projection()) - with _running(project) as (first, log): - orphan = _until(lambda: _active_copy(project), "First runner never started copying") - owner = _lock_owner(project) - _assert_locked_refusal(project, owner, orphan) - assert first.poll() is None - assert first.wait(timeout=180) == 0, log.read_text() - assert _checksum(project) == before - assert _heads(project) == ["bbbb"] - assert len(_helpers(project)) == 1 - - -@pytest.mark.parametrize("case", ["distributed", "partition", "mutation", "engine", "async0"]) -def test_rebuild_preflight_refuses_without_helpers(project, case): - _seed(project, 10000) - target = _target(project) - if case == "distributed": - cluster = project.client.command("SELECT any(cluster) FROM system.clusters") - assert cluster, "Owned ClickHouse fixture has no configured cluster for a Distributed table" - project.client.command( - f"RENAME TABLE {project.database}.events TO {project.database}.local_events" - ) - project.client.command( - f"CREATE TABLE {project.database}.events (id UInt64, ts DateTime, k UInt8, payload String) " - f"ENGINE = Distributed('{cluster}', '{project.database}', 'local_events', id)" - ) - elif case == "partition": - target = target.replace("PARTITION BY toYYYYMM(ts)", "PARTITION BY toYYYYMM(ts) + 1") - elif case == "mutation": - project.stop_merges(f"{project.database}.events") - project.client.command( - f"ALTER TABLE {project.database}.events UPDATE k = k + 1 WHERE id = 1" - ) - _until(lambda: _pending_mutation(project), "Mutation did not become pending") - elif case == "engine": - target = target.replace("ENGINE = MergeTree()", "ENGINE = Memory") - else: - _send_async0(project, "preflight-" + secrets.token_hex(6), 900001) - _until(lambda: _queued_async(project), "Async wait0 insert was not queued") - try: - _revision(project, target=target) - rejected = project.run("up", "it", "--timeout", "5") - assert rejected.exit_code != 0, rejected.output - assert _heads(project) == ["aaaa"] - assert not _helpers(project) - assert not _locks(project) - finally: - if case == "mutation": - project.client.command(f"SYSTEM START MERGES {project.database}.events") - if case == "async0": - project.drain_async_queue("events") - - -def test_rebuild_replicated_database_updates_both_replicas( - cluster_project, clickhouse_cluster, request -): - cluster = clickhouse_cluster - database = "it_rep_" + secrets.token_hex(8) - for node, client in cluster.clients.items(): - request.addfinalizer( - lambda owned=client: owned.command(f"DROP DATABASE IF EXISTS {database} SYNC") - ) - client.command( - f"CREATE DATABASE {database} ENGINE = Replicated('/clickhouse/databases/{database}', '01', 'r{node}') " - "SETTINGS collection_name = 'chm_it_auth'" - ) - config_path = cluster_project.root / "config.yaml" - config = yaml.safe_load(config_path.read_text()) - config["environments"]["it"]["database"] = database - config["environments"]["it"].pop("cluster") - config_path.write_text(yaml.safe_dump(config)) - project = replace(cluster_project, database=database) - project.client.command( - f"CREATE TABLE {database}.events (id UInt64, ts DateTime, k UInt8, payload String) " - "ENGINE = ReplicatedMergeTree PARTITION BY toYYYYMM(ts) ORDER BY id" - ) - _baseline(project) - project.client.command( - f"INSERT INTO {database}.events SELECT number, toDateTime('2026-01-01') + number, " - "number % 7, toString(number) FROM numbers(4000)" - ) - cluster.clients[2].command(f"SYSTEM SYNC REPLICA {database}.events") - before = _checksum(project) - _revision(project, target=_target(project, engine="ReplicatedMergeTree")) - for client in cluster.clients.values(): - client.command("SYSTEM FLUSH LOGS") - result = project.run("up", "it", "--timeout", "180") - assert result.exit_code == 0, result.output - for client in cluster.clients.values(): - client.command(f"SYSTEM SYNC REPLICA {database}.events") - assert client.command(f"SELECT count() FROM {database}.events") == before[0] - assert ( - client.command(f"SELECT sum(cityHash64(id, ts, k, payload)) FROM {database}.events") - == before[1] - ) - assert "ORDER BY (k, ts, id)" in client.command(f"SHOW CREATE TABLE {database}.events") - assert client.query( - "SELECT total_replicas FROM system.replicas WHERE database = {db:String} AND table = 'events'", - parameters={"db": database}, - ).result_rows == [(2,)] - report = _rebuild_result(project) - assert len(report["old_uuids"]) == len(report["new_uuids"]) == 2 - - -@pytest.mark.parametrize("shared_source_uuid", [True, False]) -def test_rebuild_configured_atomic_cluster_waits_for_both_replicas( - cluster_project, clickhouse_cluster, shared_source_uuid -): - project, cluster = cluster_project, clickhouse_cluster - table = f"{project.database}.events" - schema = ( - "(id UInt64, ts DateTime, k UInt8, payload String) " - f"ENGINE = ReplicatedMergeTree('/chm-rebuild-test/{project.database}/events', '{{replica}}') " - "PARTITION BY toYYYYMM(ts) ORDER BY id" - ) - if shared_source_uuid: - project.client.command(f"CREATE TABLE {table} ON CLUSTER {cluster.name} {schema}") - else: - for client in cluster.clients.values(): - client.command(f"CREATE TABLE {table} UUID '{uuid4()}' {schema}") - _baseline(project) - project.client.command( - f"INSERT INTO {table} SELECT number, toDateTime('2026-01-01'), number % 7, toString(number) " - "FROM numbers(3000)" - ) - for client in cluster.clients.values(): - client.command(f"SYSTEM SYNC REPLICA {table}") - client.command("SYSTEM FLUSH LOGS") - before = _checksum(project) - explicit_engine = ( - f"ReplicatedMergeTree('/chm-rebuild-test/{project.database}/events', '{{replica}}')" - ) - _revision(project, target=_target(project, engine=explicit_engine)) - if shared_source_uuid: - config_path = project.root / "config.yaml" - original_config = config_path.read_text() - unscoped = yaml.safe_load(original_config) - unscoped["environments"]["it"].pop("cluster") - config_path.write_text(yaml.safe_dump(unscoped)) - refused = project.run("up", "it", "--timeout", "10") - assert refused.exit_code != 0 and "incomplete_replica_scope" in refused.output - assert not _locks(project) and not _helpers(project) - config_path.write_text(original_config) - applied = project.run("up", "it", "--timeout", "90") - assert applied.exit_code == 0, applied.output - for client in cluster.clients.values(): - assert ( - client.query( - f"SELECT count(), sum(cityHash64(id, ts, k, payload)) FROM {table}" - ).result_rows[0] - == before - ) - assert "ORDER BY (k, ts, id)" in client.command(f"SHOW CREATE TABLE {table}") - assert len(_rebuild_result(project)["new_uuids"]) == 2 - assert len(set(_rebuild_result(project)["old_uuids"].values())) == ( - 1 if shared_source_uuid else 2 - ) - rollback = _rebuild_result(project)["rollback_table"] - project.client.command( - f"DROP TABLE {project.database}.{rollback} ON CLUSTER {cluster.name} SYNC" - ) - project.client.command(f"INSERT INTO {table} VALUES (4001, '2026-01-02', 2, 'after cleanup')") - for client in cluster.clients.values(): - client.command(f"SYSTEM SYNC REPLICA {table}") - assert client.command(f"SELECT count() FROM {table}") == before[0] + 1 - - -def test_rebuild_after_exchange_resumes_cleanup_only(project, clickhouse_server): - before = _seed(project, 2000) - old_uuid = _uuid(project) - _revision(project) - prefix = f"DROP TABLE `{project.database}`.`events__chm_dual`" - with _hold_command(project, clickhouse_server, prefix) as (blocked, release): - with _running(project) as (process, log): - assert blocked.wait(30), log.read_text() - rebuilt_uuid = _uuid(project) - assert rebuilt_uuid != old_uuid - owner = _lock_owner(project) - process.kill() - assert process.wait(timeout=15) != 0 - release.set() - _until( - lambda: project.client.command( - "SELECT count() FROM system.tables WHERE database={db:String} AND name='events__chm_dual'", - parameters={"db": project.database}, - ) - == 0, - "Owned DROP did not complete", - ) - assert process.poll() is not None - assert _lock_owner(project) == owner - assert _heads(project) == ["aaaa"] - project.client.command(f"DROP TABLE {project.database}._chm_rebuild_lock_events SYNC") - resumed = project.run("up", "it", "--timeout", "30") - assert resumed.exit_code == 0, resumed.output + log.read_text() - assert _uuid(project) == rebuilt_uuid - assert _checksum(project) == before - assert _helpers(project) == [_rebuild_result(project)["rollback_table"]] - assert _heads(project) == ["bbbb"] - - -def test_rows_stranded_in_old_storage_stop_for_the_operator(project, clickhouse_server): - """A write that reaches only the old storage after the swap stops the run until released.""" - before = _seed(project, 2000) - _revision(project) - db, rollback = project.database, "events__chm_old_bbbb" - insert = f"INSERT INTO {db}.{rollback} VALUES (9000000, '2026-02-02', 1, 'stranded')" - with _hold_command(project, clickhouse_server, "SELECT query_id, sum(rows) FROM") as gate: - blocked, release = gate - with _running(project) as (process, log): - # Cloud reaches the check only after its async-log reads settle (40 s seen). - assert blocked.wait(180), log.read_text() - # Stands in for an insert that resolved the old table just before EXCHANGE. - project.client.command(insert) - project.flush_logs() - release.set() - assert process.wait(timeout=90) != 0 - assert "reached only" in log.read_text(), log.read_text() - assert _heads(project) == ["aaaa"] and _locks(project) - refused = project.run("up", "it", "--timeout", project.budget(5)) - assert refused.exit_code != 0, refused.output - project.client.command( - f"INSERT INTO {db}.events SELECT * FROM {db}.{rollback} WHERE id = 9000000" - ) - project.client.command(f"DROP TABLE {db}._chm_rebuild_lock_events SYNC") - resumed = project.run("up", "it", "--timeout", project.budget(30)) - assert resumed.exit_code == 0, resumed.output - stranded = _rebuild_result(project)["stranded"] - assert stranded["rows"] == 1 and stranded["acknowledged"] - assert stranded["query_ids"] == [ - project.client.command( - # part_log names the flush query when the server buffers the insert asynchronously. - f"SELECT any(query_id) FROM {project.query_log()} WHERE type = 'QueryFinish' " - "AND query_kind IN ('Insert', 'AsyncInsertFlush') " - "AND position(query, {table:String}) > 0", - parameters={"table": f"{db}.{rollback}"}, - ) - ] - assert _checksum(project)[0] == before[0] + 1 - assert ( - project.client.command(f"SELECT payload FROM {db}.events WHERE id = 9000000") == "stranded" - ) - assert _heads(project) == ["bbbb"] - - -def test_rebuild_async0_opt_in_reports_old_uuid_flush_errors(project, clickhouse_server): - before = _seed(project, 20000) - old_uuid = _uuid(project) - _revision(project, allow_async_loss=True) - query_id = "async0-" + secrets.token_hex(8) - # Hold the real HTTP EXCHANGE request (after the pre-swap flush where one is allowed). - # Enqueue exactly one old-UUID insert in that gap, then let ClickHouse swap. - with _hold_command(project, clickhouse_server, "EXCHANGE TABLES") as (blocked, release): - with _running(project) as (process, log): - assert blocked.wait(30), log.read_text() - try: - _send_async0(project, query_id, 1000000) - assert _queued_async(project) - finally: - release.set() - assert process.wait(timeout=90) == 0, log.read_text() - report = _rebuild_result(project)["async_flush_errors"] - project.flush_logs() - errors = project.client.query( - f"SELECT query_id, bytes FROM {project.everywhere('system.asynchronous_insert_log')} " - "WHERE database = {db:String} AND table = 'events' AND status = 'FlushError' " - "AND position(exception, {uuid:String}) > 0", - parameters={"db": project.database, "uuid": old_uuid}, - ).result_rows - assert len(errors) == 1 and errors[0][0] == query_id - assert (report["count"], report["bytes"], report["query_ids"]) == (1, errors[0][1], [query_id]) - assert report["complete"] is not project.cloud - assert _checksum(project) == before - assert _heads(project) == ["bbbb"] - - -def test_plan_understands_guarded_rebuild_without_running_it(project): - before = _seed(project, 1000) - source_uuid = _uuid(project) - _revision(project) - planned = project.run("plan", "it", "--json") - assert planned.exit_code == 0, planned.output - document = json.loads(planned.stdout) - assert document["gate_would_refuse"] is False - statement = document["migrations"][0]["statements"][0] - assert statement["classification"]["kind"] == "rebuild" - assert statement["file"] == "migrations/sql/rebuild.sql" - assert statement["rebuild"]["part_count"] == 1 - assert _uuid(project) == source_uuid - assert _checksum(project) == before - assert not _helpers(project) and not _locks(project) - - -@pytest.mark.parametrize("masked", [False, True]) -def test_rebuild_preserves_empty_logical_tables(project, masked): - _seed(project, 30 if masked else 0) - if masked: - project.client.command( - f"DELETE FROM {project.database}.events WHERE 1 SETTINGS lightweight_deletes_sync=1" - ) - assert _checksum(project)[0] == 0 - source_uuid = _uuid(project) - _revision(project) - applied = project.run("up", "it") - assert applied.exit_code == 0, applied.output - assert _uuid(project) != source_uuid - assert _checksum(project)[0] == 0 - assert "ORDER BY (k, ts, id)" in project.client.command( - f"SHOW CREATE TABLE {project.database}.events" - ) - assert _heads(project) == ["bbbb"] - - -def test_completed_rebuild_is_not_repeated_after_later_interruption(project): - before = _seed(project, 2000) - _revision(project) - revision = project.versions_dir / "bbbb.py" - revision.write_text( - revision.read_text().replace( - "def downgrade():", " op.execute('SELECT sleep(2)')\n\ndef downgrade():" - ) - ) - with _running(project) as (process, log): - _until( - lambda: project.client.command( - f"SELECT count() FROM {project.everywhere('system.processes')} " - "WHERE startsWith(query, 'SELECT sleep(2)')" - ), - "Rebuild did not reach its following read", - seconds=120, - ) - assert not _locks(project) - rebuilt_uuid = _uuid(project) - process.kill() - assert process.wait(timeout=15) != 0 - assert _heads(project) == ["aaaa"] - resumed = project.run("up", "it") - assert resumed.exit_code == 0, resumed.output + log.read_text() - assert _uuid(project) == rebuilt_uuid - assert _checksum(project) == before - assert len(_helpers(project)) == 1 - assert _heads(project) == ["bbbb"] - - -def test_rebuild_survives_helper_merges_resuming_mid_copy(owned_project): - """A server restart re-enables merges; snapshot helpers must still keep every counted row.""" - project = owned_project - db = project.database - project.client.command( - f"CREATE TABLE {db}.events (id UInt64, ts DateTime, k UInt8, payload String) " - "ENGINE = ReplacingMergeTree PARTITION BY toYYYYMM(ts) ORDER BY id" - ) - _baseline(project) - project.client.command(f"SYSTEM STOP MERGES {db}.events") # Keep duplicate rows to snapshot. - for _ in range(2): - project.client.command( - f"INSERT INTO {db}.events SELECT number, toDateTime('2026-02-01') + number, " - "number % 7, toString(number) FROM numbers(60000)" - ) - project.client.command("SYSTEM FLUSH LOGS") - before = project.client.query( - f"SELECT count(), sum(cityHash64(id, ts, k, payload)) FROM {db}.events FINAL" - ).result_rows[0] - # Slow enough to act mid-copy, within ClickHouse's 3 s sleep-per-block limit. - slow = "id + toUInt64(sleepEachRow(0.0001)) AS id, ts, k, payload" - _revision(project, target=_target(project, "ReplacingMergeTree"), select=slow) - try: - with _running(project, timeout="90") as (process, log): - _until(lambda: _active_copy(project), "Copy never started", seconds=60) - for role in ("snap", "stage"): - helper = f"{db}.events__chm_{role}" - project.client.command(f"SYSTEM START MERGES {helper}") - project.client.command(f"OPTIMIZE TABLE {helper} FINAL") - assert process.wait(timeout=120) == 0, log.read_text() - finally: - project.client.command(f"SYSTEM START MERGES {db}.events") - after = project.client.query( - f"SELECT count(), sum(cityHash64(id, ts, k, payload)) FROM {db}.events FINAL" - ).result_rows[0] - assert after == before - assert _heads(project) == ["bbbb"] - - -def test_rebuild_survives_replica_restart_mid_copy(cluster_project, clickhouse_cluster): - """A real restart of the other replica mid-copy: merges resume there, and nothing is lost.""" - project, cluster = cluster_project, clickhouse_cluster - table = f"{project.database}.events" - engine = ( - f"ReplicatedReplacingMergeTree('/chm-restart/{project.database}/events', '{{replica}}')" - ) - project.client.command( - f"CREATE TABLE {table} ON CLUSTER {cluster.name} " - f"(id UInt64, ts DateTime, k UInt8, payload String) ENGINE = {engine} " - "PARTITION BY toYYYYMM(ts) ORDER BY id" - ) - _baseline(project) - for client in cluster.clients.values(): - client.command(f"SYSTEM STOP MERGES {table}") # Keep duplicate rows to snapshot. - for _ in range(2): - project.client.command( - f"INSERT INTO {table} SELECT number, toDateTime('2026-02-01') + number, " - "number % 7, toString(number) FROM numbers(60000)" - ) - for client in cluster.clients.values(): - client.command(f"SYSTEM SYNC REPLICA {table}") - client.command("SYSTEM FLUSH LOGS") - before = _final_checksum(project.client, table) - target = _target( - project, f"ReplicatedReplacingMergeTree('/chm-restart/{{db}}/new', '{{replica}}')" - ) - slow = "id + toUInt64(sleepEachRow(0.0001)) AS id, ts, k, payload" - _revision(project, target=target, select=slow) - try: - _restart_replica_mid_copy(project, cluster) - finally: - for client in cluster.clients.values(): - client.command(f"SYSTEM START MERGES {table}") - assert _heads(project) == ["bbbb"] - for client in cluster.clients.values(): - client.command(f"SYSTEM SYNC REPLICA {table}") - assert _final_checksum(client, table) == before - - -def _restart_replica_mid_copy(project, cluster): - with _running(project, timeout="180") as (process, log): - _until(lambda: _active_copy(project), "Copy never started", seconds=60) - cluster.stop_node(2) - cluster.start_node(2) # Node 2's STOP MERGES on the helpers, if any, is now gone. - exit_code = process.wait(timeout=180) - if exit_code != 0: # A rebuild may stop while node 2 is down; the next `up` resumes it. - resumed = project.run("up", "it", "--timeout", "180") - assert resumed.exit_code == 0, log.read_text() + resumed.output - - -def _final_checksum(client, table): - return client.query( - f"SELECT count(), sum(cityHash64(id, ts, k, payload)) FROM {table} FINAL" - ).result_rows[0] - - -def _baseline(project): - project.write_revision("aaaa", {"upgrade": 'op.execute("SELECT 1")'}) - result = project.run("up", "it") - assert result.exit_code == 0, result.output - - -def _seed(project, rows, first_partition_rows=0): - db = project.database - project.client.command( - f"CREATE TABLE {db}.events (id UInt64, ts DateTime, k UInt8, payload String) " - "ENGINE = MergeTree() PARTITION BY toYYYYMM(ts) ORDER BY id" - ) - _baseline(project) - if first_partition_rows: - project.client.command( - f"INSERT INTO {db}.events SELECT number, toDateTime('2026-01-01') + number, " - f"number % 7, toString(number) FROM numbers({first_partition_rows})" - ) - project.client.command( - f"INSERT INTO {db}.events SELECT 100000 + number, toDateTime('2026-02-01') + number, " - f"number % 7, toString(number) FROM numbers({rows})" - ) - project.flush_logs() # Observer seeds complete writer evidence. - return _checksum(project) - - -def _target(project, engine="MergeTree()"): - return ( - "CREATE TABLE {db}.events (id UInt64, ts DateTime, k UInt8, payload String) " - f"ENGINE = {engine} PARTITION BY toYYYYMM(ts) ORDER BY (k, ts, id)" - ) - - -def _revision(project, *, target=None, select=None, allow_async_loss=False): - (project.sql_dir / "rebuild.sql").write_text(target or _target(project)) - options = [] - if select is not None: - options.append(f"select={select!r}") - if allow_async_loss: - options.append("allow_unacknowledged_async_loss=True") - if project.cloud: - options.append("writers_on_this_service_only=True") # The test's writers use one service. - suffix = ", " + ", ".join(options) if options else "" - project.write_revision( - "bbbb", {"upgrade": f"op.rebuild_table('events', 'rebuild.sql'{suffix})"}, "aaaa" - ) - - -def _slow_projection(): - return "id + toUInt64(sleepEachRow(0.0003)) AS id, ts, k, payload" - - -def _checksum(project, name="events"): - return project.client.query( - f"SELECT count(), sum(cityHash64(id, ts, k, payload)) FROM {project.database}.{name}" - ).result_rows[0] - - -def _partition_rows(project, name, partition): - return project.client.command( - f"SELECT count() FROM {project.database}.{name} WHERE _partition_id = {{part:String}}", - parameters={"part": partition}, - ) - - -def _heads(project): - return [ - row[0] - for row in project.client.query( - f"SELECT version_num FROM {project.database}.alembic_version ORDER BY version_num" - ).result_rows - ] - - -def _helpers(project): - return [ - row[0] - for row in project.client.query( - "SELECT name FROM system.tables WHERE database = {db:String} " - "AND startsWith(name, 'events__chm_') ORDER BY name", - parameters={"db": project.database}, - ).result_rows - ] - - -def _locks(project): - return project.client.query( - "SELECT name, toString(uuid) FROM system.tables WHERE database = {db:String} " - "AND name = '_chm_rebuild_lock_events'", - parameters={"db": project.database}, - ).result_rows - - -def _lock_owner(project): - locks = _locks(project) - assert len(locks) == 1, locks - rows = project.client.query( - f"SELECT owner, max(heartbeat) FROM {project.database}._chm_rebuild_lock_events GROUP BY owner" - ).result_rows - assert len(rows) == 1, rows - return {"owner": rows[0][0], "table_uuid": locks[0][1]} - - -def _lock_age(project): - return project.client.command( - f"SELECT dateDiff('second', max(heartbeat), now()) FROM {project.database}._chm_rebuild_lock_events" - ) - - -def _assert_locked_refusal(project, owner, orphan): - original_uuid = _uuid(project) - second = project.run("up", "it", "--timeout", project.budget(1)) - assert second.exit_code != 0, second.output - assert "lock" in second.output.lower(), second.output - assert "reconcil" in second.output.lower(), second.output - assert _lock_owner(project) == owner - assert _uuid(project) == original_uuid - if _active_copy(project) != orphan: - project.flush_logs() - outcome = project.client.command( - f"SELECT type FROM {project.query_log()} WHERE query_id = {{id:String}} " - "ORDER BY event_time_microseconds DESC LIMIT 1", - parameters={"id": orphan}, - ) - assert outcome == "QueryFinish", "Newcomer must not cancel the old copy" - - -def _operator_release(project, process, owner): - assert process.poll() is not None, "Operator must first prove prior process cannot resume" - assert _lock_owner(project) == owner, "Operator must reconcile unchanged lock ownership" - assert _heads(project) == ["aaaa"], "Operator must reconcile migration journal/version" - assert _active_copy(project), "Operator must account for server-side in-flight work" - project.client.command(f"DROP TABLE {project.database}._chm_rebuild_lock_events SYNC") - assert not _locks(project) - - -def _copy_after_moved_partition(project, partition): - query_id = _active_copy(project) - if query_id and _partition_rows(project, "events__chm_new", partition) == 200: - return query_id - return None - - -def _pending_mutation(project): - return project.client.command( - "SELECT count() FROM system.mutations WHERE database = {db:String} " - "AND table = 'events' AND is_done = 0", - parameters={"db": project.database}, - ) - - -def _queued_async(project): - return project.client.command( - f"SELECT count() FROM {project.everywhere('system.asynchronous_inserts')} " - "WHERE database = {db:String} AND table = 'events' AND length(entries.query_id) > 0", - parameters={"db": project.database}, - ) - - -def _active_copy(project): - rows = project.client.query( - f"SELECT query_id FROM {project.everywhere('system.processes')} " - "WHERE startsWith(query_id, 'chm-rebuild-') " - "AND query_kind = 'Insert' AND position(query, {db:String}) > 0", - parameters={"db": project.database}, - ).result_rows - return rows[0][0] if rows else None - - -def _uuid(project): - return project.client.command( - "SELECT toString(uuid) FROM system.tables WHERE database = {db:String} AND name = 'events'", - parameters={"db": project.database}, - ) - - -def _send_async0(project, query_id, row_id, client=None): - sql = ( - f"INSERT INTO {project.database}.events (id, ts, k, payload) VALUES " - f"({row_id}, '2026-02-02 00:00:00', 1, 'async')" - ) - settings = { - "query_id": query_id, - "async_insert": 1, - "wait_for_async_insert": 0, - "async_insert_use_adaptive_busy_timeout": 0, - "async_insert_busy_timeout_ms": 8000, - "async_insert_busy_timeout_max_ms": 8000, - } - if not project.cloud: # Cloud locks log_query_settings (already on there). - settings["log_query_settings"] = 1 - (client or project.client).command(sql, settings=settings) - - -def _rebuild_result(project): - rows = project.client.query( - f"SELECT argMax(payload, sequence) FROM {project.database}._ch_migrate_journal " - "WHERE revision = 'bbbb' AND position > 0 GROUP BY generation, position" - ).result_rows - records = [json.loads(row[0]) for row in rows] - matching = [record for record in records if record.get("kind") == "rebuild"] - assert len(matching) == 1, records - assert matching[0]["phase"] == "done", matching[0] - return matching[0]["result"] - - -def _until(predicate, message, seconds=30): - deadline = time.monotonic() + seconds - while time.monotonic() < deadline: - result = predicate() - if result: - return result - time.sleep(0.05) - pytest.fail(message) - - -@contextmanager -def _running(project, timeout="180"): - path = project.root / f"rebuild-{time.monotonic_ns()}.log" - with path.open("w") as log: - process = subprocess.Popen( - [sys.executable, "-m", "ch_migrate.cli", "up", "it", "--timeout", timeout], - cwd=project.root, - stdout=log, - stderr=subprocess.STDOUT, - env=os.environ.copy(), - ) - try: - yield process, path - finally: - if process.poll() is None: - process.kill() - process.wait(timeout=15) - - -@contextmanager -def _hold_command(project, upstream, sql_prefix): - blocked, release = threading.Event(), threading.Event() - forwarder = _Upstream(upstream) - - class Forwarder(BaseHTTPRequestHandler): - protocol_version = "HTTP/1.1" - - def do_POST(self): - body = _request_body(self) - query = parse_qs(urlsplit(self.path).query).get("query", [""])[0] - if (query + body.decode("utf-8", errors="ignore")).lstrip().startswith(sql_prefix): - blocked.set() - if not release.wait(30): - self.send_error(504, "Test exchange barrier timed out") - return - headers = { - key: value - for key, value in self.headers.items() - if key.lower() not in {"host", "connection", "transfer-encoding", "content-length"} - } - headers["Content-Length"] = str(len(body)) - try: - status, response_headers, payload = forwarder.forward( - self.command, self.path, (body, headers) - ) - self.send_response(status) - for key, value in response_headers: - if key.lower() not in {"content-length", "transfer-encoding", "connection"}: - self.send_header(key, value) - self.send_header("Content-Length", str(len(payload))) - self.send_header("Connection", "close") - self.end_headers() - self.wfile.write(payload) - self.close_connection = True - except (BrokenPipeError, ConnectionResetError): - pass # A deliberate SIGKILL can close the downstream client after server success. - - do_GET = do_POST - - def log_message(self, *args): - pass # Never log authentication-bearing request query strings. - - server = ThreadingHTTPServer(("127.0.0.1", 0), Forwarder) - thread = threading.Thread(target=server.serve_forever, daemon=True) - config_path = project.root / "config.yaml" - original = config_path.read_text() - config = yaml.safe_load(original) - config["environments"]["it"].update(host="127.0.0.1", port=server.server_port, secure=False) - config_path.write_text(yaml.safe_dump(config)) - thread.start() - try: - yield blocked, release - finally: - release.set() - server.shutdown() - server.server_close() - thread.join(timeout=10) - forwarder.close() - config_path.write_text(original) - - -class _Upstream: - """Forward to the real server, keeping one upstream connection per HTTP session. - - ClickHouse Cloud routes each connection, not each session id, to a replica, so a - fresh connection per request could reach a replica without the session. - """ - - def __init__(self, server): - self.server = server - self.lock = threading.Lock() - self.sessions = {} - - def forward(self, method, path, request): - session = parse_qs(urlsplit(path).query).get("session_id", [None])[0] - if session is None: - connection = self._connect() - try: - return _exchange(connection, method, path, request) - finally: - connection.close() - with self.lock: - entry = self.sessions.setdefault(session, {"lock": threading.Lock(), "conn": None}) - with entry["lock"]: - entry["conn"] = entry["conn"] or self._connect() - try: - return _exchange(entry["conn"], method, path, request) - except (http.client.HTTPException, OSError): - entry["conn"].close() - entry["conn"] = None - raise - - def close(self): - for entry in self.sessions.values(): - if entry["conn"] is not None: - entry["conn"].close() - - def _connect(self): - if self.server.secure: - return http.client.HTTPSConnection( - self.server.host, - self.server.port, - timeout=120, - context=ssl.create_default_context(cafile=certifi.where()), - ) - return http.client.HTTPConnection(self.server.host, self.server.port, timeout=120) - - -def _exchange(connection, method, path, request): - body, headers = request - connection.request(method, path, body=body, headers=headers) - response = connection.getresponse() - return response.status, response.getheaders(), response.read() - - -def _request_body(handler): - if handler.headers.get("Transfer-Encoding", "").lower() != "chunked": - return handler.rfile.read(int(handler.headers.get("Content-Length", 0))) - body = bytearray() - while True: - size = int(handler.rfile.readline().split(b";", 1)[0], 16) - if size == 0: - while handler.rfile.readline().strip(): - pass # Consume optional HTTP trailers. - return bytes(body) - body.extend(handler.rfile.read(size)) - assert handler.rfile.read(2) == b"\r\n" diff --git a/tests/integration/test_rebuild_dependents.py b/tests/integration/test_rebuild_dependents.py deleted file mode 100644 index 06150ba..0000000 --- a/tests/integration/test_rebuild_dependents.py +++ /dev/null @@ -1,498 +0,0 @@ -"""Real-server dependent behavior during an online table rebuild, self-hosted and on Cloud.""" - -from __future__ import annotations - -import json -import os -import subprocess -import sys -import threading -import time -from contextlib import contextmanager -from uuid import uuid4 - -import pytest -from test_rebuild_commands import _hold_command - -pytestmark = pytest.mark.integration - - -def test_live_aggregate_and_dictionary_survive_rebuild(project, clickhouse_server, request): - _source(project) - _aggregate(project) - _dictionary(project, request) - _baseline(project) - _seed(project) - db = project.database - if project.cloud: # Cloud loads dictionaries at CREATE, before the seed rows existed. - project.client.command(f"SYSTEM RELOAD DICTIONARY ON CLUSTER 'default' {db}.events_dict") - initial = _totals(project) - assert initial == _aggregate_totals(project) - assert project.client.command(f"SELECT dictGetUInt64('{db}.events_dict', 'value', 1)") == 1 - assert project.client.command(f"SELECT dictHas('{db}.events_dict', 2000000)") == 0 - _revision(project) - acknowledged = [] - with _writer(clickhouse_server, project, acknowledged) as errors: - _until(lambda: len(acknowledged) >= 16, "Writer never acknowledged its first batch") - prefix = f"ALTER TABLE `{db}`.`events__chm_snap` ATTACH PARTITION" - with _hold_command(project, clickhouse_server, prefix) as (blocked, release): - with _running(project) as (process, log): - assert blocked.wait(30), log.read_text() - acknowledged_before = len(acknowledged) - _until( - lambda: len(acknowledged) > acknowledged_before, - "No insert committed inside the held snapshot window", - ) - forced_ids = { - row_id - for _, batch in acknowledged[acknowledged_before:] - for row_id, _, _ in batch - } - release.set() - assert process.wait(timeout=180) == 0, log.read_text() - assert not errors, errors - assert acknowledged - result = _result(project) - duplicates = _assert_accounting(project, acknowledged, initial, result["duplicate_window"]) - assert forced_ids and forced_ids.issubset(duplicates) - assert project.client.command(f"SELECT dictHas('{db}.events_dict', 2000000)") == 1 - assert ( - project.client.command(f"SELECT dictGetUInt64('{db}.events_dict', 'value', 2000000)") == 1 - ) - aggregate_before = _aggregate_totals(project) - source_before = _totals(project) - project.client.command(f"INSERT INTO {db}.events VALUES (8000000, '2026-02-02', 1, 43)") - assert _aggregate_totals(project)[1] == aggregate_before[1] + 43 - assert _totals(project)[1] == source_before[1] + 43 - assert _heads(project) == ["bbbb"] - - -def test_dictionary_reload_reads_the_replacement_storage(project, clickhouse_server, request): - _source(project) - _dictionary(project, request) - _baseline(project) - _seed(project, 100) - db = project.database - old_uuid = _uuid(project) - assert project.client.command(f"SELECT dictHas('{db}.events_dict', 9000000)") == 0 - _revision(project) - with _hold_command(project, clickhouse_server, "SYSTEM RELOAD DICTIONARY") as ( - blocked, - release, - ): - with _running(project) as (process, log): - # On Cloud the grace pause and all-replica barriers come before the reload. - assert blocked.wait(float(project.budget(30))), log.read_text() - assert _uuid(project) != old_uuid - try: - project.client.command( - f"INSERT INTO {db}.events VALUES (9000000, '2026-02-02', 1, 777)" - ) - assert ( - project.client.command( - f"SELECT count() FROM {db}.events__chm_old_bbbb WHERE id=9000000" - ) - == 0 - ) - finally: - release.set() - assert process.wait(timeout=90) == 0, log.read_text() - assert ( - project.client.command(f"SELECT dictGetUInt64('{db}.events_dict', 'value', 9000000)") == 777 - ) - assert _heads(project) == ["bbbb"] - - -@pytest.mark.parametrize("change", ["drop", "rename", "type"]) -def test_invalid_source_view_refuses_then_repaired_sql_succeeds(project, change): - _source(project) - _aggregate(project) - _baseline(project) - _seed(project, 200) - before = (_uuid(project), _source_rows(project), _aggregate_totals(project)) - projection = { - "drop": "id, ts, k", - "rename": "id, ts, k, value AS renamed_value", - "type": "id, ts, k, toString(value) AS value", - }[change] - _revision(project, _invalid_target(change), projection) - rejected = project.run("up", "it", "--timeout", project.budget(15)) - assert rejected.exit_code != 0, rejected.output - assert "events_mv" in rejected.output, rejected.output - assert (_uuid(project), _source_rows(project), _aggregate_totals(project)) == before - assert not _helpers(project) and not _locks(project) - assert _heads(project) == ["aaaa"] - _revision(project) - repaired = project.run("up", "it", "--timeout", "180") - assert repaired.exit_code == 0, repaired.output - assert _uuid(project) != before[0] - assert _source_rows(project) == before[1] - assert _aggregate_totals(project) == before[2] - assert _heads(project) == ["bbbb"] - - -@pytest.mark.parametrize("qualifier", ["none", "table", "database_table"]) -def test_inner_engine_materialized_view_keeps_receiving_rows(project, qualifier): - _source(project) - db = project.database - prefix = {"none": "", "table": "events.", "database_table": f"{db}.events."}[qualifier] - project.client.command( - f"CREATE MATERIALIZED VIEW {db}.inner_mv ENGINE = SummingMergeTree() ORDER BY k " - f"AS SELECT {prefix}k AS k, sum({prefix}value) AS value FROM {db}.events GROUP BY {prefix}k" - ) - _baseline(project) - _seed(project, 200) - before = _inner_totals(project) - assert before == _totals(project) - _revision(project) - applied = project.run("up", "it", "--timeout", "180") - assert applied.exit_code == 0, applied.output - assert _inner_totals(project) == before - project.client.command(f"INSERT INTO {db}.events VALUES (500000, '2026-02-02', 1, 37)") - assert _inner_totals(project) == _totals(project) - assert _inner_totals(project)[1] == before[1] + 37 - - -@pytest.mark.parametrize("feed_id_type", ["UInt64", "UInt32"]) -def test_materialized_view_to_rebuilt_table_uses_new_target(project, feed_id_type): - _source(project) - db = project.database - project.client.command( - f"CREATE TABLE {db}.feed (id {feed_id_type}, ts DateTime, k UInt8, value UInt64) " - "ENGINE = MergeTree() ORDER BY id" - ) - project.client.command( - f"CREATE MATERIALIZED VIEW {db}.feed_mv TO {db}.events " - f"AS SELECT id, ts, k, value FROM {db}.feed" - ) - _baseline(project) - project.client.command(f"INSERT INTO {db}.feed VALUES (1, '2026-02-02', 1, 17)") - assert _source_rows(project) == [(1, 1, 17)] - before_uuid = _uuid(project) - _revision(project) - applied = project.run("up", "it", "--timeout", "180") - assert applied.exit_code == 0, applied.output - assert _uuid(project) != before_uuid - project.client.command(f"INSERT INTO {db}.feed VALUES (2, '2026-02-02', 2, 31)") - assert _source_rows(project) == [(1, 1, 17), (2, 2, 31)] - assert _heads(project) == ["bbbb"] - - -def test_single_shard_distributed_route_is_conservatively_refused(project): - db = project.database - # Cloud's only cluster is one shard with several replicas; the refusal is per route. - cluster = project.client.query( - "SELECT cluster FROM system.clusters GROUP BY cluster " - "HAVING uniqExact(shard_num) = 1 ORDER BY count(), cluster LIMIT 1" - ).result_rows - assert cluster, "Owned server has no single-shard Distributed cluster" - _source(project) - _baseline(project) - project.client.command( - f"CREATE TABLE {db}.events_router (id UInt64, ts DateTime, k UInt8, value UInt64) " - f"ENGINE = Distributed('{cluster[0][0]}', '{db}', 'events', id)" - ) - project.client.command(f"INSERT INTO {db}.events VALUES (1, '2026-02-02', 1, 9)") - before = (_uuid(project), _source_rows(project)) - _revision(project) - rejected = project.run("up", "it", "--timeout", project.budget(15)) - assert rejected.exit_code != 0, rejected.output - assert "Distributed" in rejected.output - assert (_uuid(project), _source_rows(project)) == before - assert not _helpers(project) and not _locks(project) - assert _heads(project) == ["aaaa"] - - -def _source(project): - project.client.command( - f"CREATE TABLE {project.database}.events " - "(id UInt64, ts DateTime, k UInt8, value UInt64) " - "ENGINE = MergeTree() PARTITION BY toYYYYMM(ts) ORDER BY id" - ) - - -def _aggregate(project): - db = project.database - project.client.command( - f"CREATE TABLE {db}.totals (k UInt8, value UInt64, rows UInt64) " - "ENGINE = SummingMergeTree() ORDER BY k" - ) - project.client.command( - f"CREATE MATERIALIZED VIEW {db}.events_mv TO {db}.totals AS " - f"SELECT k, sum(value) AS value, count() AS rows FROM {db}.events GROUP BY k" - ) - - -def _dictionary(project, request): - db = project.database - reader = "reader_" + db - if project.cloud: # Cloud needs a password, and a dictionary source must name it. - password = project.new_password() - identity, credentials = f"BY '{password}'", f" PASSWORD '{password}'" - identity = "IDENTIFIED WITH sha256_password " + identity - else: - identity, credentials = "IDENTIFIED WITH no_password HOST LOCAL", "" - project.client.command(f"CREATE USER {reader} {identity}") - request.addfinalizer(lambda: project.client.command(f"DROP USER IF EXISTS {reader}")) - project.client.command(f"GRANT SELECT ON {db}.events TO {reader}") - project.client.command( - f"CREATE DICTIONARY {db}.events_dict (id UInt64, value UInt64) " - f"PRIMARY KEY id SOURCE(CLICKHOUSE(USER '{reader}'{credentials} DB '{db}' TABLE 'events')) " - "LIFETIME(MIN 0 MAX 0) LAYOUT(HASHED())" - ) - - -def _baseline(project): - project.write_revision("aaaa", {"upgrade": 'op.execute("SELECT 1")'}) - result = project.run("up", "it") - assert result.exit_code == 0, result.output - - -def _seed(project, rows=12000): - project.client.command( - f"INSERT INTO {project.database}.events " - "SELECT number + 1, toDateTime('2026-02-01'), number % 7, " - f"number % 23 + 1 FROM numbers({rows})" - ) - project.flush_logs() - - -def _revision(project, target=None, select=None): - (project.sql_dir / "rebuild.sql").write_text(target or _target()) - projection = select or "id, ts, k, value" - declaration = ", writers_on_this_service_only=True" if project.cloud else "" - project.write_revision( - "bbbb", - { - "upgrade": f"op.rebuild_table('events', 'rebuild.sql', select={projection!r}{declaration})" - }, - "aaaa", - ) - - -def _target(): - return ( - "CREATE TABLE {db}.events (id UInt64, ts DateTime, k UInt8, value UInt64) " - "ENGINE = MergeTree() PARTITION BY toYYYYMM(ts) ORDER BY (k, ts, id)" - ) - - -def _invalid_target(change): - columns = { - "drop": "id UInt64, ts DateTime, k UInt8", - "rename": "id UInt64, ts DateTime, k UInt8, renamed_value UInt64", - "type": "id UInt64, ts DateTime, k UInt8, value String", - } - return ( - f"CREATE TABLE {{db}}.events ({columns[change]}) ENGINE = MergeTree() " - "PARTITION BY toYYYYMM(ts) ORDER BY (k, ts, id)" - ) - - -def _uuid(project): - return project.client.command( - "SELECT toString(uuid) FROM system.tables " - "WHERE database = {db:String} AND name = 'events'", - parameters={"db": project.database}, - ) - - -def _source_rows(project): - return project.client.query( - f"SELECT id, k, value FROM {project.database}.events ORDER BY id, k, value" - ).result_rows - - -def _totals(project): - return dict( - project.client.query( - f"SELECT k, sum(value) FROM {project.database}.events GROUP BY k ORDER BY k" - ).result_rows - ) - - -def _aggregate_totals(project): - return dict( - project.client.query( - f"SELECT k, sum(value) FROM {project.database}.totals GROUP BY k ORDER BY k" - ).result_rows - ) - - -def _aggregate_counts(project): - return dict( - project.client.query( - f"SELECT k, sum(rows) FROM {project.database}.totals GROUP BY k ORDER BY k" - ).result_rows - ) - - -def _inner_totals(project): - return dict( - project.client.query( - f"SELECT k, sum(value) FROM {project.database}.inner_mv GROUP BY k ORDER BY k" - ).result_rows - ) - - -def _helpers(project): - return project.client.query( - "SELECT name FROM system.tables WHERE database = {db:String} " - "AND startsWith(name, 'events__chm_') ORDER BY name", - parameters={"db": project.database}, - ).result_rows - - -def _locks(project): - return project.client.query( - "SELECT name FROM system.tables WHERE database = {db:String} " - "AND name = '_chm_rebuild_lock_events'", - parameters={"db": project.database}, - ).result_rows - - -def _heads(project): - return [ - row[0] - for row in project.client.query( - f"SELECT version_num FROM {project.database}.alembic_version ORDER BY version_num" - ).result_rows - ] - - -def _result(project): - rows = project.client.query( - f"SELECT argMax(payload, sequence) FROM {project.database}._ch_migrate_journal " - "WHERE revision = 'bbbb' AND position > 0 GROUP BY generation, position" - ).result_rows - records = [json.loads(row[0]) for row in rows] - matching = [item for item in records if item.get("kind") == "rebuild"] - assert len(matching) == 1, records - assert matching[0]["phase"] == "done", matching[0] - return matching[0]["result"] - - -def _assert_accounting(project, acknowledged, initial, window): - expected = {row_id: (key, value) for _, batch in acknowledged for row_id, key, value in batch} - assert len(expected) == sum(len(batch) for _, batch in acknowledged) - actual = project.client.query( - f"SELECT id, any(k), any(value), count() FROM {project.database}.events " - "WHERE id >= 2000000 GROUP BY id" - ).result_rows - assert {row_id: (key, value) for row_id, key, value, _ in actual} == expected - assert all(count in (1, 2) for _, _, _, count in actual) - duplicates = {row_id: (value, count - 1) for row_id, _, value, count in actual if count > 1} - intervals = _server_intervals(project, [query_id for query_id, _ in acknowledged], window) - owners = {row_id: query_id for query_id, batch in acknowledged for row_id, _, _ in batch} - assert all(intervals[owners[row_id]] for row_id in duplicates) - excess_by_key = {key: 0 for key in initial} - for row_id, (value, repeats) in duplicates.items(): - key = expected[row_id][0] - excess_by_key[key] += value * repeats - expected_once = dict(initial) - expected_counts = dict( - project.client.query( - f"SELECT k, count() FROM {project.database}.{_result(project)['rollback_table']} " - "WHERE id < 2000000 GROUP BY k" - ).result_rows - ) - for _, (key, value) in expected.items(): - expected_once[key] = expected_once.get(key, 0) + value - expected_counts[key] = expected_counts.get(key, 0) + 1 - aggregate = _aggregate_totals(project) - assert aggregate == expected_once - assert _aggregate_counts(project) == expected_counts - assert {key: _totals(project)[key] - aggregate[key] for key in aggregate} == excess_by_key - return duplicates - - -def _server_intervals(project, query_ids, window): - project.flush_logs() - rows = project.client.query( - "SELECT query_id, countIf(type = 'QueryStart'), countIf(type = 'QueryFinish'), " - "minIf(event_time_microseconds, type = 'QueryStart') <= " - "parseDateTime64BestEffort({end:String}, 6) AND " - "maxIf(event_time_microseconds, type = 'QueryFinish') >= " - "parseDateTime64BestEffort({start:String}, 6) AS overlaps " - f"FROM {project.query_log()} WHERE has({{ids:Array(String)}}, query_id) " - "AND type IN ('QueryStart', 'QueryFinish') GROUP BY query_id", - parameters={"ids": query_ids, "start": window["start"], "end": window["end"]}, - ).result_rows - evidence = { - query_id: (starts, finishes, bool(overlap)) for query_id, starts, finishes, overlap in rows - } - assert set(evidence) == set(query_ids) - assert all(starts == 1 and finishes == 1 for starts, finishes, _ in evidence.values()) - return {query_id: overlap for query_id, (_, _, overlap) in evidence.items()} - - -def _until(predicate, message, seconds=30): - deadline = time.monotonic() + seconds - while time.monotonic() < deadline: - if predicate(): - return - time.sleep(0.05) - pytest.fail(message) - - -@contextmanager -def _writer(server, project, acknowledged): - stop = threading.Event() - errors = [] - - def send(): - client = server.connect() - try: - batch_number = 0 - while not stop.is_set(): - batch = [(2000000 + batch_number * 16 + n, n % 7, n % 23 + 1) for n in range(16)] - values = ", ".join( - f"({row_id}, '2026-02-02', {key}, {value})" for row_id, key, value in batch - ) - query_id = f"chm-deps-writer-{uuid4().hex}" - try: - client.command( - f"INSERT INTO {project.database}.events (id, ts, k, value) VALUES {values}", - settings={ - "query_id": query_id, - "log_queries": 1, - "async_insert": 0, - "wait_for_async_insert": 1, - }, - ) - except Exception as exc: - errors.append(exc) - break - acknowledged.append((query_id, batch)) - batch_number += 1 - stop.wait(0.03) - finally: - client.close() - - thread = threading.Thread(target=send, daemon=True) - thread.start() - try: - yield errors - finally: - stop.set() - thread.join(timeout=30) - assert not thread.is_alive(), "Synchronous writer did not stop" - - -@contextmanager -def _running(project): - path = project.root / f"dependent-rebuild-{uuid4().hex}.log" - with path.open("w") as output: - process = subprocess.Popen( - [sys.executable, "-m", "ch_migrate.cli", "up", "it", "--timeout", "180"], - cwd=project.root, - env=os.environ.copy(), - stdout=output, - stderr=subprocess.STDOUT, - ) - try: - yield process, path - finally: - if process.poll() is None: - process.kill() - process.wait(timeout=15) diff --git a/tests/integration/test_rebuild_preflight_live.py b/tests/integration/test_rebuild_preflight_live.py deleted file mode 100644 index 06d06f7..0000000 --- a/tests/integration/test_rebuild_preflight_live.py +++ /dev/null @@ -1,125 +0,0 @@ -"""Read-only preflight against owned ClickHouse writer profiles and live mutations.""" - -import secrets -from dataclasses import replace - -import clickhouse_connect -import pytest - -from ch_migrate.introspect import get_live_schema -from ch_migrate.rebuild_preflight import RebuildRequest, inspect_rebuild - -pytestmark = pytest.mark.integration - - -def test_preflight_accepts_synchronous_default_writer(project): - request = _source(project) - project.client.command(f"INSERT INTO {project.database}.events VALUES (1, 2)") - project.client.command("SYSTEM FLUSH LOGS") - result = inspect_rebuild(project.client, request) - refusals = {item.code for item in result.findings if item.severity == "refusal"} - # Cloud refuses until the migration declares single-service writers; nothing else may refuse. - assert refusals == ({"cloud_writers_undeclared"} if project.cloud else set()) - assert result.part_count == 1 - assert result.partition_parts == {"all": 1} - assert result.insert_rows_per_second == pytest.approx(1 / 3600) - - -def test_preflight_detects_inherited_fire_and_forget_profile(project, clickhouse_server): - if project.cloud: - pytest.skip("ClickHouse Cloud forbids changing log_query_settings in a settings profile") - request = _source(project) - name = "writer_" + secrets.token_hex(6) - password = secrets.token_hex(24) - project.client.command( - f"CREATE SETTINGS PROFILE {name} SETTINGS async_insert = 1, wait_for_async_insert = 0, log_query_settings = 0" - ) - project.client.command(f"CREATE USER {name} IDENTIFIED BY '{password}' SETTINGS PROFILE {name}") - try: - project.client.command(f"GRANT INSERT ON {project.database}.events TO {name}") - writer = clickhouse_connect.get_client( - host=clickhouse_server.host, - port=clickhouse_server.port, - username=name, - password=password, - ) - try: - writer.command(f"INSERT INTO {project.database}.events VALUES (1, 2)") - finally: - writer.close() - project.client.command("SYSTEM FLUSH LOGS") - settings = project.client.query( - "SELECT Settings FROM system.query_log WHERE user = {user:String} AND query_kind = 'Insert'", - parameters={"user": name}, - ).result_rows - assert settings and all(not row[0] for row in settings) - result = inspect_rebuild(project.client, request) - risky = [item for item in result.findings if item.code == "unacknowledged_async_writer"] - assert len(risky) == 1 - assert risky[0].severity == "refusal" - assert risky[0].details["user"] == name - allowed = inspect_rebuild( - project.client, replace(request, allow_unacknowledged_async_loss=True) - ) - assert all( - item.severity == "warning" - for item in allowed.findings - if item.code == "unacknowledged_async_writer" - ) - finally: - project.client.command(f"DROP USER IF EXISTS {name}") - project.client.command(f"DROP SETTINGS PROFILE IF EXISTS {name}") - - -def test_preflight_checks_both_replicas_and_distributed_routes(cluster_project, clickhouse_cluster): - project, cluster = cluster_project, clickhouse_cluster - table = f"{project.database}.events" - project.client.command( - f"CREATE TABLE {table} ON CLUSTER {cluster.name} (id UInt64, value UInt64) " - "ENGINE = ReplicatedMergeTree('/chm-preflight/{database}/{table}', '{replica}') ORDER BY id" - ) - cluster.clients[2].command(f"INSERT INTO {table} VALUES (1, 2)") - project.client.command(f"SYSTEM SYNC REPLICA {table}") - for client in cluster.clients.values(): - client.command("SYSTEM FLUSH LOGS") - source = get_live_schema(project.client, project.database).tables["events"] - request = RebuildRequest(project.database, source, replace(source, name="new"), cluster.name) - result = inspect_rebuild(project.client, request) - assert not [item for item in result.findings if item.severity == "refusal"] - assert result.part_count == 1 # Replica copies are not two logical parts. - assert result.insert_rows_per_second == pytest.approx(1 / 3600) - project.client.command( - f"CREATE TABLE {project.database}.route ON CLUSTER {cluster.name} AS {table} " - f"ENGINE = Distributed('{cluster.name}', '{project.database}', 'events')" - ) - routed = inspect_rebuild(project.client, request) - routes = [item for item in routed.findings if item.code == "distributed_route"] - assert len({item.details["host"] for item in routes}) == 2 - assert all(item.severity == "refusal" for item in routes) - - -def test_preflight_refuses_pending_mutation_and_partition_change(project): - request = _source(project) - table = f"{project.database}.events" - project.client.command(f"INSERT INTO {table} VALUES (1, 2)") - project.stop_merges(table) - try: - project.client.command( - f"ALTER TABLE {table} UPDATE value = value + 1 WHERE 1 SETTINGS mutations_sync = 0" - ) - target = replace(request.target, partition_by="id") - result = inspect_rebuild(project.client, replace(request, target=target)) - codes = {item.code for item in result.findings if item.severity == "refusal"} - assert {"unfinished_mutations", "partition_key_change"} <= codes - finally: - project.client.command(f"SYSTEM START MERGES {table}") - - -def _source(project): - project.client.command( - f"CREATE TABLE {project.database}.events (id UInt64, value UInt64) ENGINE = MergeTree ORDER BY id" - ) - source = get_live_schema(project.client, project.database).tables["events"] - return RebuildRequest( - project.database, source, replace(source, name="replacement", order_by=["value"]) - ) diff --git a/tests/integration/test_scaffold_rebuild.py b/tests/integration/test_scaffold_rebuild.py deleted file mode 100644 index 3099444..0000000 --- a/tests/integration/test_scaffold_rebuild.py +++ /dev/null @@ -1,106 +0,0 @@ -"""Fetch live DDL, edit only SQL, plan it and execute the generated guarded revision.""" - -import json -import shlex -from pathlib import Path - -import pytest -from jsonschema import Draft202012Validator - -pytestmark = pytest.mark.integration -SCHEMA = Path(__file__).parents[2] / "docs" / "schemas" / "plan.schema.json" - - -def test_scaffold_rebuild_round_trip_uses_only_sql_edits(project): - table = f"{project.database}.t" - project.client.command( - f"CREATE TABLE {table} (id UInt64, k UInt8, value String) ENGINE = MergeTree ORDER BY id" - ) - project.client.command( - f"INSERT INTO {table} SELECT number, number % 7, toString(number) FROM numbers(1000)" - ) - before = project.client.query( - f"SELECT count(), sum(cityHash64(id, k, value)) FROM {table}" - ).result_rows[0] - project.write_revision("aaaa", {"upgrade": "op.execute('SELECT 1')"}) - baseline = project.run("up", "it") - assert baseline.exit_code == 0, baseline.output - created = project.run("new", "it", "reorder", "--table", "t", "--rebuild") - assert created.exit_code == 0, created.output - [sql_file] = list((project.sql_dir / "history" / "tables" / "t").glob("*.up.sql")) - assert list(project.sql_dir.rglob("*.down.sql")) == [] - source = sql_file.read_text() - assert "{db}.`t`" in source - assert project.database not in source - assert "ORDER BY id" in source - sql_file.write_text(source.replace("ORDER BY id", "ORDER BY (k, id)", 1)) - project.client.command("SYSTEM FLUSH LOGS") - planned = project.run("plan", "it", "--json") - assert planned.exit_code == 0, planned.output - document = json.loads(planned.stdout) - Draft202012Validator(json.loads(SCHEMA.read_text())).validate(document) - statement = document["migrations"][0]["statements"][0] - assert statement["classification"]["kind"] == "rebuild" - assert shlex.split(statement["suggested_command"]) == [ - "ch-migrate", - "new", - "it", - "rebuild_t", - "--table", - "t", - "--rebuild", - ] - assert document["schema_version"] == 1 - human = project.run("plan", "it") - assert human.exit_code == 0, human.output - assert statement["suggested_command"] in human.output - refusals = { - item["code"] for item in statement["rebuild"]["findings"] if item["severity"] == "refusal" - } - # The scaffold names the Cloud declaration as False; Cloud refuses until it is True. - assert refusals == ({"cloud_writers_undeclared"} if project.cloud else set()) - if project.cloud: - _assert_cloud_refusal(project, project.run("up", "it", "--timeout", "30"), before) - [revision] = project.versions_dir.glob("*reorder*.py") - revision.write_text( - revision.read_text().replace( - "writers_on_this_service_only=False", "writers_on_this_service_only=True" - ) - ) - applied = project.run("up", "it", "--timeout", "180") - assert applied.exit_code == 0, applied.output - assert "ORDER BY (k, id)" in project.client.command(f"SHOW CREATE TABLE {table}") - assert ( - project.client.query( - f"SELECT count(), sum(cityHash64(id, k, value)) FROM {table}" - ).result_rows[0] - == before - ) - refused = project.run("down", "it") - assert refused.exit_code != 0 - assert "irreversible" in refused.output.lower() - assert ( - project.client.query( - f"SELECT count(), sum(cityHash64(id, k, value)) FROM {table}" - ).result_rows[0] - == before - ) - - -def _assert_cloud_refusal(project, applied, before): - """Undeclared Cloud writers: refused before any helper, lock or swap exists.""" - assert applied.exit_code != 0, applied.output - assert "cloud_writers_undeclared" in applied.output - table = f"{project.database}.t" - assert "ORDER BY id" in project.client.command(f"SHOW CREATE TABLE {table}") - assert ( - project.client.query( - f"SELECT count(), sum(cityHash64(id, k, value)) FROM {table}" - ).result_rows[0] - == before - ) - leftovers = project.client.query( - "SELECT name FROM system.tables WHERE database = {db:String} AND name LIKE '%chm%'", - parameters={"db": project.database}, - ).result_rows - assert leftovers == [] diff --git a/tests/integration/test_version_table.py b/tests/integration/test_version_table.py index 7a80492..dddb7da 100644 --- a/tests/integration/test_version_table.py +++ b/tests/integration/test_version_table.py @@ -24,20 +24,11 @@ def test_version_table_interrupted_delete_does_not_replay(project): process = _start_up(project) try: _wait_for(lambda: set(_heads(project)) == {"aaaa", "bbbb"}) - original_ids = _wait_for_version_delete(project, process) + _wait_for_version_delete(project, process) assert process.poll() is None, "up must still be waiting for the version DELETE" _stop_process(process) - refused = _completed_up(project, ("--timeout", "0.4")) - assert refused[0] != 0, refused - current_ids = { - row[0] - for row in project.client.query( - "SELECT mutation_id FROM system.mutations WHERE database = {db:String} " - "AND table = 'alembic_version'", - parameters={"db": project.database}, - ).result_rows - } - assert current_ids == original_ids + refused = _completed_up(project) + assert refused[0] != 0 and "overlap" in refused[1].lower(), refused assert _probe_counts(project) == [("aaaa", 1), ("bbbb", 1)] finally: _stop_process(process) @@ -47,6 +38,7 @@ def test_version_table_interrupted_delete_does_not_replay(project): assert completed[0] == 0, completed[1] assert _probe_counts(project) == [("aaaa", 1), ("bbbb", 1), ("cccc", 1)] assert _heads(project) == ["cccc"] + _assert_version_sql(project) def test_version_table_failed_mutation_refuses_without_killing(project): @@ -211,9 +203,9 @@ def _probe_revision(project, revision, parent): ) -def _start_up(project, extra=()): +def _start_up(project): return subprocess.Popen( - [sys.executable, "-m", "ch_migrate.cli", "up", "it", *extra], + [sys.executable, "-m", "ch_migrate.cli", "up", "it"], cwd=project.root, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, @@ -232,8 +224,8 @@ def _stop_process(process): return process.communicate(timeout=5)[0] -def _completed_up(project, extra=()): - process = _start_up(project, extra) +def _completed_up(project): + process = _start_up(project) try: output = process.communicate(timeout=20)[0] return process.returncode, output @@ -276,7 +268,7 @@ def _wait_for_version_delete(project, process): parameters={"db": project.database}, ).result_rows if any(not done and "DELETE" in command.upper() for _, command, done, _ in rows): - return {mutation for mutation, _, _, _ in rows} + return if process.poll() is not None: break time.sleep(0.05) @@ -294,6 +286,25 @@ def _failed_mutations(project, client=None): ).result_rows +def _assert_version_sql(project): + project.client.command("SYSTEM FLUSH LOGS") + queries = [ + row[0] + for row in project.client.query( + "SELECT DISTINCT replaceAll(query, '`', '') FROM system.query_log WHERE type = 'QueryStart' " + "AND position(query, {db:String}) > 0 AND position(query, 'alembic_version') > 0", + parameters={"db": project.database}, + ).result_rows + ] + prefix = f"{project.database}.alembic_version" + assert any(query.startswith(f"INSERT INTO {prefix}") for query in queries) + deletes = [query for query in queries if query.startswith(f"ALTER TABLE {prefix} DELETE")] + assert deletes and all( + re.search(r"mutations_sync\s*=\s*2", query) for query in deletes + ), queries + assert not any(query.startswith(f"ALTER TABLE {prefix} UPDATE") for query in queries) + + def _bootstrap_cluster_user(project, cluster, request): monkeypatch = request.getfixturevalue("monkeypatch") user = project.database + "_migration" diff --git a/tests/integration/waiting_spike_probe.py b/tests/integration/waiting_spike_probe.py deleted file mode 100644 index 7865f00..0000000 --- a/tests/integration/waiting_spike_probe.py +++ /dev/null @@ -1,558 +0,0 @@ -"""Explicit local experiments for the waiting design; not auto-collected.""" - -import secrets -import subprocess -import sys -import time -from pathlib import Path - -import clickhouse_connect -import pytest - -pytestmark = pytest.mark.integration - - -def test_waiting_metadata_and_mutation_markers(project): - for table in ("mutations", "distributed_ddl_queue", "query_log", "part_log"): - columns = project.client.query( - "SELECT name, type FROM system.columns WHERE database = 'system' AND table = {table:String} " - "AND (position(name, 'query') > 0 OR position(name, 'mutation') > 0 OR name IN ('command', 'entry', 'settings')) " - "ORDER BY position", - parameters={"table": table}, - ).result_rows - print(table, columns) - target = f"{project.database}.ownership" - project.client.command( - f"CREATE TABLE {target} (id UInt64, x UInt64) ENGINE = MergeTree ORDER BY id" - ) - project.client.command(f"INSERT INTO {target} VALUES (1, 0), (2, 0)") - project.client.command(f"SYSTEM STOP MERGES {target}") - try: - statements = [ - f"ALTER TABLE {target} UPDATE x = x + 1 WHERE id = 1 /* chm_marker_comment */", - f"ALTER TABLE {target} UPDATE x = x + 1 WHERE id = 2 AND 'chm_marker_expression' = 'chm_marker_expression'", - ] - for sql in statements: - query_id = "chm_probe_" + secrets.token_hex(8) - project.client.command( - sql, settings={"mutations_sync": 0, "query_id": query_id, "log_comment": query_id} - ) - print("submitted", query_id) - rows = project.client.query( - "SELECT mutation_id, command, is_done FROM system.mutations " - "WHERE database = {db:String} AND table = 'ownership' ORDER BY mutation_id", - parameters={"db": project.database}, - ).result_rows - print("mutation commands", rows) - project.client.command("SYSTEM FLUSH LOGS") - events = project.client.query( - "SELECT query_id, type, log_comment, query FROM system.query_log " - "WHERE startsWith(query_id, 'chm_probe_') AND position(query, {db:String}) > 0 " - "ORDER BY event_time_microseconds", - parameters={"db": project.database}, - ).result_rows - profiles = project.client.query( - "SELECT query_id, mapFilter((k, v) -> positionCaseInsensitive(k, 'mutation') > 0, ProfileEvents) " - "FROM system.query_log WHERE type = 'QueryFinish' AND startsWith(query_id, 'chm_probe_') " - "AND position(query, {db:String}) > 0", - parameters={"db": project.database}, - ).result_rows - print("mutation profile events", profiles) - print("submission events", events) - finally: - project.client.command(f"SYSTEM START MERGES {target}") - deadline = time.monotonic() + 10 - while time.monotonic() < deadline: - values = project.client.query(f"SELECT id, x FROM {target} ORDER BY id").result_rows - if values == [(1, 1), (2, 1)]: - break - time.sleep(0.05) - assert values == [(1, 1), (2, 1)] - - -def test_waiting_schema_mutation_marker_options(project, clickhouse_server, request): - clauses = [ - "MODIFY COLUMN x UInt32, DELETE WHERE 0 AND 'chm_schema_marker' = 'chm_schema_marker'", - "MATERIALIZE COLUMN y, DELETE WHERE 0 AND 'chm_schema_marker' = 'chm_schema_marker'", - "CLEAR COLUMN y, DELETE WHERE 0 AND 'chm_schema_marker' = 'chm_schema_marker'", - "RENAME COLUMN y TO renamed, DELETE WHERE 0 AND 'chm_schema_marker' = 'chm_schema_marker'", - ] - for index, clause in enumerate(clauses): - name = f"schema_probe_{index}" - table = f"{project.database}.{name}" - project.client.command( - f"CREATE TABLE {table} (id UInt64, x UInt64, y UInt64 DEFAULT x + 1) " - "ENGINE = MergeTree ORDER BY id" - ) - project.client.command(f"INSERT INTO {table} (id, x) VALUES (1, 10), (2, 20)") - project.client.command(f"SYSTEM STOP MERGES {table}") - server = clickhouse_server - worker = clickhouse_connect.get_client( - host=server.host, - port=server.port, - username=server.user, - password=server.password, - secure=server.secure, - send_receive_timeout=2, - ) - request.addfinalizer(worker.close) - query_id = "chm_schema_" + secrets.token_hex(8) - try: - try: - worker.command( - f"ALTER TABLE {table} {clause}", - settings={"mutations_sync": 0, "alter_sync": 0, "query_id": query_id}, - ) - print("combined marker request", clause, "returned") - except Exception as error: - print( - "combined marker request", - clause, - type(error).__name__, - str(error).splitlines()[0], - ) - finally: - project.client.command(f"SYSTEM START MERGES {table}") - deadline = time.monotonic() + 10 - while time.monotonic() < deadline: - active = project.client.query( - "SELECT count() FROM system.processes WHERE query_id = {id:String}", - parameters={"id": query_id}, - ).result_rows[0][0] - if not active: - break - time.sleep(0.05) - result = project.client.query( - "SELECT mutation_id, command, is_done FROM system.mutations WHERE database = {db:String} AND table = {table:String}", - parameters={"db": project.database, "table": name}, - ).result_rows - print("combined marker durable commands", clause, result) - - -def test_waiting_ddl_settings_identity(cluster_project, clickhouse_cluster): - project, cluster = cluster_project, clickhouse_cluster - marker = "chm_ddl_" + secrets.token_hex(8) - cluster.stop_node(2) - try: - cluster.clients[1].command( - f"CREATE TABLE {project.database}.ddl_owned ON CLUSTER {cluster.name} (id UInt64) ENGINE = Memory", - settings={ - "distributed_ddl_task_timeout": 0, - "distributed_ddl_output_mode": "none", - "log_comment": marker, - }, - ) - rows = ( - cluster.clients[1] - .query( - "SELECT entry, host, query, settings['log_comment'], status " - "FROM system.distributed_ddl_queue WHERE position(query, {db:String}) > 0 " - "AND position(query, 'ddl_owned') > 0 ORDER BY host", - parameters={"db": project.database}, - ) - .result_rows - ) - print("DDL durable marker", marker, rows) - assert len(rows) == 2 and all(row[3] == marker for row in rows) - finally: - cluster.start_node(2) - - -def test_waiting_actual_mutation_corpus(project): - cases = [ - ("update", "ALTER TABLE {table} UPDATE x = x + 1 WHERE id = 1"), - ("delete", "ALTER TABLE {table} DELETE WHERE id = 1"), - ("lightweight_delete", "DELETE FROM {table} WHERE id = 1"), - ("lightweight_update", "UPDATE {table} SET x = x + 1 WHERE id = 1"), - ("modify_type", "ALTER TABLE {table} MODIFY COLUMN x UInt32"), - ("drop_column", "ALTER TABLE {table} DROP COLUMN y"), - ("clear_column", "ALTER TABLE {table} CLEAR COLUMN y"), - ("materialize_column", "ALTER TABLE {table} MATERIALIZE COLUMN y"), - ("rename_column", "ALTER TABLE {table} RENAME COLUMN y TO z"), - ("modify_ttl", "ALTER TABLE {table} MODIFY TTL ts + INTERVAL 1 DAY"), - ("failure", "ALTER TABLE {table} UPDATE x = throwIf(x = 5) WHERE 1"), - ] - for label, template in cases: - name = "corpus_" + label - table = f"{project.database}.{name}" - project.client.command( - f"CREATE TABLE {table} (id UInt64, x UInt64, y UInt64 DEFAULT x + 1, ts DateTime DEFAULT now()) " - "ENGINE = MergeTree ORDER BY id" - ) - project.client.command(f"INSERT INTO {table} (id, x) VALUES (1, 1), (5, 5)") - settings = {"mutations_sync": 0, "alter_sync": 0, "lightweight_deletes_sync": 0} - if label == "lightweight_update": - settings["allow_experimental_lightweight_update"] = 1 - try: - project.client.command(template.format(table=table), settings=settings) - outcome = "accepted" - except Exception as error: - outcome = type(error).__name__ + ": " + str(error).splitlines()[0] - deadline = time.monotonic() + 5 - while True: - mutations = project.client.query( - "SELECT mutation_id, command, is_done, parts_to_do, latest_fail_reason " - "FROM system.mutations WHERE database = {db:String} AND table = {table:String}", - parameters={"db": project.database, "table": name}, - ).result_rows - if ( - not mutations - or all(row[2] or row[4] for row in mutations) - or time.monotonic() > deadline - ): - break - time.sleep(0.05) - print("mutation corpus", label, outcome, mutations) - - -def test_waiting_single_node_deduplicated_intent(project): - table = f"{project.database}.intents" - project.client.command( - f"CREATE TABLE {table} (step String, owner String) ENGINE = MergeTree ORDER BY step " - "SETTINGS non_replicated_deduplication_window = 10000" - ) - for owner in ("first", "second"): - project.client.insert( - table, - [["step1", owner]], - column_names=["step", "owner"], - settings={"insert_deduplication_token": "same-step", "insert_deduplicate": 1}, - ) - rows = project.client.query(f"SELECT step, owner FROM {table}").result_rows - print("single intent dedup", rows) - assert rows == [("step1", "first")] - - -def test_waiting_cluster_deduplicated_intent(cluster_project, clickhouse_cluster): - from concurrent.futures import ThreadPoolExecutor - from threading import Barrier - - project, cluster = cluster_project, clickhouse_cluster - table = f"{project.database}.intents" - cluster.clients[1].command( - f"CREATE TABLE {table} ON CLUSTER {cluster.name} (step String, owner String) " - f"ENGINE = ReplicatedMergeTree('/clickhouse/intents/{project.database}', '{{replica}}') ORDER BY step" - ) - barrier = Barrier(2) - - def insert_owner(node): - barrier.wait() - cluster.clients[node].insert( - table, - [["step1", f"node{node}"]], - column_names=["step", "owner"], - settings={ - "insert_deduplication_token": "same-step", - "insert_deduplicate": 1, - "insert_quorum": "auto", - }, - ) - - with ThreadPoolExecutor(max_workers=2) as pool: - list(pool.map(insert_owner, (1, 2))) - for node, client in cluster.clients.items(): - client.command(f"SYSTEM SYNC REPLICA {table} LIGHTWEIGHT") - rows = client.query(f"SELECT step, owner FROM {table}").result_rows - print("cluster intent dedup", node, rows) - assert len(rows) == 1 and rows[0][1] in ("node1", "node2") - - -def test_waiting_lightweight_update_with_required_storage(project): - table = f"{project.database}.patch_probe" - project.client.command( - f"CREATE TABLE {table} (id UInt64, x UInt64) ENGINE = MergeTree ORDER BY id " - "SETTINGS enable_block_number_column = 1, enable_block_offset_column = 1" - ) - project.client.command(f"INSERT INTO {table} VALUES (1, 10), (2, 20)") - project.client.command( - f"UPDATE {table} SET x = x + 1 WHERE id = 1", - settings={"allow_experimental_lightweight_update": 1, "mutations_sync": 0}, - ) - mutations = project.client.query( - "SELECT mutation_id, command, is_done FROM system.mutations WHERE database = {db:String} AND table = 'patch_probe'", - parameters={"db": project.database}, - ).result_rows - values = project.client.query(f"SELECT id, x FROM {table} ORDER BY id").result_rows - print("lightweight update enabled", mutations, values) - assert values == [(1, 11), (2, 20)] - - -def test_waiting_empty_fence_cost_and_retention(project): - table = f"{project.database}.fence_cost" - project.client.command( - f"CREATE TABLE {table} (id UInt64, x UInt64, payload String) ENGINE = MergeTree ORDER BY id" - ) - project.client.command( - f"INSERT INTO {table} SELECT number, number, repeat('payload', 100) FROM numbers(10000)" - ) - project.client.command( - f"ALTER TABLE {table} DELETE WHERE 0 AND 'chm_cost_marker' = 'chm_cost_marker'", - settings={"mutations_sync": 2}, - ) - project.client.command("SYSTEM FLUSH LOGS") - names = { - row[0] - for row in project.client.query( - "SELECT name FROM system.columns WHERE database = 'system' AND table = 'part_log'" - ).result_rows - } - fields = [ - name - for name in ("event_type", "query_id", "rows", "read_rows", "read_bytes", "ProfileEvents") - if name in names - ] - if fields: - events = project.client.query( - "SELECT " + ", ".join(fields) + " FROM system.part_log " - "WHERE database = {db:String} AND table = 'fence_cost' AND toString(event_type) LIKE '%Mutat%'", - parameters={"db": project.database}, - ).result_rows - print("empty fence physical cost", fields, events) - mutations = project.client.query( - "SELECT command, is_done, parts_to_do FROM system.mutations WHERE database = {db:String} AND table = 'fence_cost'", - parameters={"db": project.database}, - ).result_rows - print("empty fence retained", mutations) - assert project.client.command(f"SELECT count() FROM {table}") == 10000 - - -@pytest.mark.parametrize("trial", range(3)) -def test_waiting_kill_before_receipt_and_resume_from_intent(project, trial): - table = f"{project.database}.counter" - project.client.command(f"CREATE TABLE {table} (x UInt64) ENGINE = MergeTree ORDER BY tuple()") - project.client.command(f"INSERT INTO {table} VALUES (0)") - table_uuid = str( - project.client.query( - "SELECT uuid FROM system.tables WHERE database = {db:String} AND name = 'counter'", - parameters={"db": project.database}, - ).result_rows[0][0] - ) - project.client.command( - f"CREATE TABLE {project.database}.waiting_intent (token String, table_uuid String) " - "ENGINE = MergeTree ORDER BY token SETTINGS non_replicated_deduplication_window = 10000" - ) - token = "chm_recovery_" + secrets.token_hex(8) - project.client.insert( - f"{project.database}.waiting_intent", - [[token, table_uuid]], - column_names=["token", "table_uuid"], - settings={"insert_deduplication_token": "revision-step", "insert_deduplicate": 1}, - ) - project.client.command(f"SYSTEM STOP MERGES {table}") - scripts = Path(__file__).parents[2] / "docs/design/spikes/2026-10-02-waiting" - submitted = subprocess.Popen( - [sys.executable, str(scripts / "submit_without_ack.py"), token], - cwd=project.root, - stdout=subprocess.PIPE, - stderr=subprocess.STDOUT, - text=True, - ) - resumed = None - try: - deadline = time.monotonic() + 10 - while time.monotonic() < deadline: - rows = project.client.query( - "SELECT mutation_id, command, is_done FROM system.mutations " - "WHERE database = {db:String} AND table = 'counter' ORDER BY mutation_id", - parameters={"db": project.database}, - ).result_rows - if any(token in row[1] for row in rows): - break - time.sleep(0.05) - assert len(rows) == 2 and any(token in row[1] for row in rows), rows - original_ids = [row[0] for row in rows] - print("pre-receipt crash", trial, rows) - submitted.kill() - submitted.communicate(timeout=5) - resumed = subprocess.Popen( - [sys.executable, str(scripts / "resume_fence.py")], - cwd=project.root, - stdout=subprocess.PIPE, - stderr=subprocess.STDOUT, - text=True, - ) - deadline = time.monotonic() + 10 - while not (project.root / "resumed_marker").exists() and time.monotonic() < deadline: - time.sleep(0.05) - assert (project.root / "resumed_marker").exists() - assert resumed.poll() is None - project.client.command(f"SYSTEM START MERGES {table}") - output = resumed.communicate(timeout=20)[0] - assert resumed.returncode == 0, output - print("recovered", trial, output.strip()) - ids = [ - row[0] - for row in project.client.query( - "SELECT mutation_id FROM system.mutations WHERE database = {db:String} " - "AND table = 'counter' ORDER BY mutation_id", - parameters={"db": project.database}, - ).result_rows - ] - assert ids == original_ids - assert project.client.command(f"SELECT x FROM {table}") == 1 - finally: - for process in (submitted, resumed): - if process is not None and process.poll() is None: - process.kill() - process.communicate(timeout=5) - project.client.command(f"SYSTEM START MERGES {table}") - - -def test_waiting_foreign_failure_blocks_later_mutation(project): - table = f"{project.database}.foreign_probe" - project.client.command( - f"CREATE TABLE {table} (id UInt64, x UInt64) ENGINE = MergeTree ORDER BY id" - ) - project.client.command(f"INSERT INTO {table} VALUES (1, 1), (5, 5)") - project.client.command( - f"ALTER TABLE {table} UPDATE x = throwIf(x = 5) WHERE 1", settings={"mutations_sync": 0} - ) - deadline = time.monotonic() + 5 - while time.monotonic() < deadline: - rows = project.client.query( - "SELECT mutation_id, command, is_done, latest_fail_reason FROM system.mutations " - "WHERE database = {db:String} AND table = 'foreign_probe' ORDER BY mutation_id", - parameters={"db": project.database}, - ).result_rows - if rows and rows[0][3]: - break - time.sleep(0.05) - assert rows[0][3] - project.client.command( - f"ALTER TABLE {table} UPDATE x = x + 1 WHERE id = 1", settings={"mutations_sync": 0} - ) - rows = project.client.query( - "SELECT mutation_id, command, is_done, parts_to_do, latest_fail_reason FROM system.mutations " - "WHERE database = {db:String} AND table = 'foreign_probe' ORDER BY mutation_id", - parameters={"db": project.database}, - ).result_rows - print("foreign failure and queued follower", rows) - assert len(rows) == 2 and all(not row[2] for row in rows) - assert project.client.query(f"SELECT id, x FROM {table} ORDER BY id").result_rows == [ - (1, 1), - (5, 5), - ] - - -def test_waiting_requires_every_replica(cluster_project, clickhouse_cluster): - project, cluster = cluster_project, clickhouse_cluster - table = f"{project.database}.replica_wait" - cluster.clients[1].command( - f"CREATE TABLE {table} ON CLUSTER {cluster.name} (id UInt64, x UInt64) " - f"ENGINE = ReplicatedMergeTree('/clickhouse/wait/{project.database}', '{{replica}}') ORDER BY id" - ) - cluster.clients[1].command(f"INSERT INTO {table} VALUES (1, 0)") - cluster.clients[2].command(f"SYSTEM SYNC REPLICA {table}") - for action in ("MERGES", "FETCHES", "REPLICATION QUEUES"): - cluster.clients[2].command(f"SYSTEM STOP {action} {table}") - try: - cluster.clients[1].command( - f"ALTER TABLE {table} UPDATE x = x + 1 WHERE 1", settings={"mutations_sync": 0} - ) - cluster.clients[2].command(f"SYSTEM SYNC REPLICA {table} PULL") - deadline = time.monotonic() + 10 - while time.monotonic() < deadline: - states = {} - for node, client in cluster.clients.items(): - states[node] = client.query( - "SELECT mutation_id, is_done, parts_to_do FROM system.mutations " - "WHERE database = {db:String} AND table = 'replica_wait'", - parameters={"db": project.database}, - ).result_rows - if states[1] and states[2] and states[1][0][1] and not states[2][0][1]: - break - time.sleep(0.05) - print("replica completion divergence", states) - assert states[1][0][1] == 1 and states[2][0][1] == 0 - assert states[1][0][0] == states[2][0][0] - finally: - for action in ("MERGES", "FETCHES", "REPLICATION QUEUES"): - cluster.clients[2].command(f"SYSTEM START {action} {table}") - cluster.clients[2].command(f"SYSTEM SYNC REPLICA {table}") - assert cluster.clients[2].query(f"SELECT x FROM {table}").result_rows == [(1,)] - - -def test_waiting_managed_sync_settings_override(project): - table = f"{project.database}.sync_settings" - project.client.command(f"CREATE TABLE {table} (x UInt64) ENGINE = MergeTree ORDER BY tuple()") - project.client.command(f"INSERT INTO {table} VALUES (0)") - project.client.command(f"SYSTEM STOP MERGES {table}") - try: - project.client.command( - f"ALTER TABLE {table} UPDATE x = x + 1 WHERE 1, " - "DELETE WHERE 0 AND 'chm_settings_marker' = 'chm_settings_marker' " - "SETTINGS mutations_sync = 2, mutations_sync = 0, alter_sync = 0" - ) - rows = project.client.query( - "SELECT mutation_id, command, is_done FROM system.mutations " - "WHERE database = {db:String} AND table = 'sync_settings'", - parameters={"db": project.database}, - ).result_rows - print("managed sync override", rows) - assert rows and all(row[2] == 0 for row in rows) - finally: - project.client.command(f"SYSTEM START MERGES {table}") - - -def test_waiting_completed_ownership_marker_can_expire(project): - table = f"{project.database}.retention_probe" - project.client.command( - f"CREATE TABLE {table} (x UInt64) ENGINE = MergeTree ORDER BY tuple() " - "SETTINGS finished_mutations_to_keep = 1, old_parts_lifetime = 0, " - "cleanup_delay_period = 1, cleanup_delay_period_random_add = 0" - ) - project.client.command(f"INSERT INTO {table} VALUES (0)") - project.client.command( - f"ALTER TABLE {table} UPDATE x = x + 1 WHERE " "'chm_expired_owner' = 'chm_expired_owner'", - settings={"mutations_sync": 2}, - ) - project.client.command( - f"ALTER TABLE {table} DELETE WHERE 0", - settings={"mutations_sync": 2}, - ) - deadline = time.monotonic() + 30 - while time.monotonic() < deadline: - rows = project.client.query( - "SELECT mutation_id, command FROM system.mutations " - "WHERE database = {db:String} AND table = 'retention_probe'", - parameters={"db": project.database}, - ).result_rows - if not any("chm_expired_owner" in row[1] for row in rows): - break - time.sleep(0.1) - print("expired ownership marker", rows, "x =", project.client.command(f"SELECT x FROM {table}")) - assert rows and not any("chm_expired_owner" in row[1] for row in rows) - assert project.client.command(f"SELECT x FROM {table}") == 1 - - -def test_waiting_ddl_failure_names_host(cluster_project, clickhouse_cluster): - project, cluster = cluster_project, clickhouse_cluster - table = f"{project.database}.ddl_failure" - for client in cluster.clients.values(): - client.command(f"CREATE TABLE {table} (id UInt64, x String) ENGINE = MergeTree ORDER BY id") - cluster.clients[2].command(f"ALTER TABLE {table} ADD COLUMN conflict String") - token = "chm_ddl_failure_" + secrets.token_hex(8) - cluster.clients[1].command( - f"ALTER TABLE {table} ON CLUSTER {cluster.name} ADD COLUMN conflict UInt64", - settings={ - "distributed_ddl_task_timeout": 0, - "distributed_ddl_output_mode": "none", - "log_comment": token, - }, - ) - deadline = time.monotonic() + 10 - while time.monotonic() < deadline: - rows = ( - cluster.clients[1] - .query( - "SELECT host, status, exception_code, exception_text FROM system.distributed_ddl_queue " - "WHERE settings['log_comment'] = {token:String} ORDER BY host", - parameters={"token": token}, - ) - .result_rows - ) - if len(rows) == 2 and any(row[2] for row in rows): - break - time.sleep(0.05) - print("DDL host exception", rows) - assert len(rows) == 2 and any(row[2] and row[3] for row in rows) diff --git a/tests/test_classify.py b/tests/test_classify.py deleted file mode 100644 index 8fcaf6b..0000000 --- a/tests/test_classify.py +++ /dev/null @@ -1,77 +0,0 @@ -"""Corpus classifications and uncertain or quoted SQL boundaries.""" - -from pathlib import Path - -import pytest -import yaml - -from ch_migrate.classify import classify -from ch_migrate.introspect import Schema, parse_create_table -from ch_migrate.statements import MigrationStatement - -CORPUS = yaml.safe_load((Path(__file__).parent / "corpus/classification.yaml").read_text()) -VALUES = {"db": "example", "table": "example.sample"} - - -@pytest.fixture -def schema(): - table = parse_create_table(CORPUS["table_ddl"].format(**VALUES)) - return Schema(database="example", tables={"sample": table}) - - -@pytest.mark.parametrize("case", CORPUS["statements"], ids=lambda case: case["id"]) -def test_classification_corpus(case, schema): - sql = case["sql"].format(**VALUES) - result = classify(sql, schema) - assert result.kind == case["expected"], (sql, result) - if "detail" in case: - assert case["detail"] in result.detail - if sql.startswith("ALTER"): - assert result.table == "example.sample" - - -@pytest.mark.parametrize( - "case", - [case for case in CORPUS["statements"] if "without_schema" in case], - ids=lambda case: case["id"], -) -def test_classification_without_live_schema_is_conservative(case): - result = classify(case["sql"].format(**VALUES)) - assert result.kind == case["without_schema"] - if "MODIFY COLUMN" in case["sql"]: - assert "if the type changes" in result.detail - - -def test_classification_ignores_comments_and_literal_keywords(schema): - statement = MigrationStatement( - "/* UPDATE x */ ALTER TABLE \"example\".`sample` ON CLUSTER 'local' " - "MODIFY COLUMN x COMMENT 'DROP COLUMN y, MODIFY ENGINE = Memory'", - "migration.sql", - 9, - (), - "upgrade", - ) - result = classify(statement, schema) - assert result.kind == "metadata" - assert result.table == "example.sample" - - -def test_classification_different_database_cannot_supply_a_type(schema): - result = classify("ALTER TABLE elsewhere.sample MODIFY COLUMN x UInt64", schema) - assert result.kind == "mutation" - assert "if the type changes" in result.detail - - -def test_classification_unknown_action_is_not_metadata(schema): - result = classify("ALTER TABLE example.sample SOMETHING UNRECOGNIZED", schema) - assert result.kind == "other" - - -def test_classification_preserves_identifier_case(schema): - result = classify("ALTER TABLE example.sample MODIFY ORDER BY ID", schema) - assert result.kind == "rebuild" - - -def test_classification_empty_and_read_only_have_no_table(): - assert classify("/* nothing */").kind == "other" - assert classify("SELECT 'ALTER TABLE x DELETE WHERE 1'").table is None diff --git a/tests/test_cli_output.py b/tests/test_cli_output.py index b60e729..e6e66b9 100644 --- a/tests/test_cli_output.py +++ b/tests/test_cli_output.py @@ -1,9 +1,5 @@ -"""Errors and warnings from `up`, its gate and `upgrade-env` use the `ui` markers. +"""Errors and warnings from `up`, its gate and `upgrade-env` use the `ui` markers.""" -`--json` callers must still get the error message unchanged, as plain text. -""" - -import json from types import SimpleNamespace import click @@ -11,11 +7,10 @@ from click.testing import CliRunner from ch_migrate import ui -from ch_migrate.cli import _print_gate_findings, main +from ch_migrate.cli import main +from ch_migrate.gate import print_gate_findings from ch_migrate.lint import LintResult, Severity -OLD_ENV_MESSAGE = "This project's migrations/env.py is from ch-migrate 0.x. Run `ch-migrate upgrade-env`." - @pytest.fixture def project(tmp_path, monkeypatch): @@ -36,7 +31,7 @@ def broken(): assert result.stderr == "✗ It broke.\nRun ch-migrate lint.\n" -def test_failure_message_keeps_hints_for_json(): +def test_failure_message_keeps_hints(): failure = ui.Failure("It broke.", "Run `ch-migrate lint`.") assert failure.format_message() == "It broke. Run `ch-migrate lint`." @@ -51,13 +46,6 @@ def test_old_env_refusal(project): ) -def test_old_env_refusal_json_message_is_unchanged(project): - (project / "migrations" / "env.py").write_text("# ch-migrate 0.5 env.py\n") - result = CliRunner().invoke(main, ["history", "dev", "--json"]) - assert result.exit_code == 2 - assert json.loads(result.stdout)["error"] == OLD_ENV_MESSAGE - - def test_gate_findings_use_lint_markers(capsys): report = SimpleNamespace( results=[ @@ -83,12 +71,9 @@ def test_gate_findings_use_lint_markers(capsys): ), ] ) - assert _print_gate_findings(report) is True + assert print_gate_findings(report) is True captured = capsys.readouterr() - assert ( - captured.err - == "✗ a.up.sql:2 Use IF NOT EXISTS [idempotency]\n CREATE TABLE t\n" - ) + assert captured.err == "✗ a.up.sql:2 Use IF NOT EXISTS [idempotency]\n CREATE TABLE t\n" assert captured.out == ( "! config.yaml Drops a column [destructive_changes]\n" "✓ b.up.sql:5 Idempotency waiver: copies once [idempotency]\n" diff --git a/tests/test_deps.py b/tests/test_deps.py index 36f7feb..00eb217 100644 --- a/tests/test_deps.py +++ b/tests/test_deps.py @@ -34,7 +34,9 @@ def _make_graph() -> DependencyGraph: graph.edges = [ DependencyEdge(source="events", target="hourly_events", dep_type=DepType.DATA_FLOW), DependencyEdge(source="events", target="hourly_events", dep_type=DepType.SCHEMA), - DependencyEdge(source="hourly_events", target="hourly_events_dest", dep_type=DepType.DATA_FLOW), + DependencyEdge( + source="hourly_events", target="hourly_events_dest", dep_type=DepType.DATA_FLOW + ), DependencyEdge(source="users", target="dict_users", dep_type=DepType.SCHEMA), ] return graph @@ -136,6 +138,7 @@ def test_renders_empty_graph(self): def test_shows_edge_count(self): import re as _re + output = StringIO() console = Console(file=output, force_terminal=True, width=100) graph = _make_graph() @@ -207,13 +210,15 @@ def test_handles_circular_dependency(self): @pytest.fixture def deps_runner(tmp_path: Path, monkeypatch): monkeypatch.chdir(tmp_path) - (tmp_path / "config.yaml").write_text(""" + (tmp_path / "config.yaml").write_text( + """ environments: dev: host: localhost database: testdb user: test -""") +""" + ) monkeypatch.setenv("CH_DEV_PASSWORD", "pass") return CliRunner() @@ -226,6 +231,7 @@ def test_shows_graph(self, mock_deps, mock_client, deps_runner): mock_deps.return_value = _make_graph() from ch_migrate.cli import main + result = deps_runner.invoke(main, ["deps", "dev"]) assert result.exit_code == 0 assert "Dependency Graph" in result.output @@ -240,6 +246,7 @@ def test_validate_safe_migration(self, mock_deps, mock_client, deps_runner, tmp_ sql_file.write_text("ALTER TABLE users ADD COLUMN email String") from ch_migrate.cli import main + result = deps_runner.invoke(main, ["deps", "dev", "--validate", str(sql_file)]) assert result.exit_code == 0 assert "passed" in result.output @@ -254,5 +261,6 @@ def test_validate_breaking_migration(self, mock_deps, mock_client, deps_runner, sql_file.write_text("DROP TABLE events") from ch_migrate.cli import main + result = deps_runner.invoke(main, ["deps", "dev", "--validate", str(sql_file)]) assert result.exit_code == 1 diff --git a/tests/test_diff.py b/tests/test_diff.py index 6ad6996..85926d8 100644 --- a/tests/test_diff.py +++ b/tests/test_diff.py @@ -84,40 +84,52 @@ def test_detects_added_column(self): local = Schema(database="db") live = Schema(database="db") local.tables["users"] = self._base_table() - live.tables["users"] = self._base_table(columns=[ - ColumnDefinition(name="id", type="UInt64"), - ColumnDefinition(name="name", type="String"), - ColumnDefinition(name="email", type="String"), - ]) + live.tables["users"] = self._base_table( + columns=[ + ColumnDefinition(name="id", type="UInt64"), + ColumnDefinition(name="name", type="String"), + ColumnDefinition(name="email", type="String"), + ] + ) diffs = compare_schemas(local, live) modified = [d for d in diffs if d.status == DiffStatus.MODIFIED] assert len(modified) == 1 - assert any("email" in fd.message and "in the database but not in the snapshot" in fd.message for fd in modified[0].field_diffs) + assert any( + "email" in fd.message and "in the database but not in the snapshot" in fd.message + for fd in modified[0].field_diffs + ) def test_detects_removed_column(self): local = Schema(database="db") live = Schema(database="db") - local.tables["users"] = self._base_table(columns=[ - ColumnDefinition(name="id", type="UInt64"), - ColumnDefinition(name="name", type="String"), - ColumnDefinition(name="phone", type="String"), - ]) + local.tables["users"] = self._base_table( + columns=[ + ColumnDefinition(name="id", type="UInt64"), + ColumnDefinition(name="name", type="String"), + ColumnDefinition(name="phone", type="String"), + ] + ) live.tables["users"] = self._base_table() diffs = compare_schemas(local, live) modified = [d for d in diffs if d.status == DiffStatus.MODIFIED] assert len(modified) == 1 - assert any("phone" in fd.message and "in the snapshot but not in the database" in fd.message for fd in modified[0].field_diffs) + assert any( + "phone" in fd.message and "in the snapshot but not in the database" in fd.message + for fd in modified[0].field_diffs + ) def test_detects_column_type_change(self): local = Schema(database="db") live = Schema(database="db") local.tables["users"] = self._base_table() - live.tables["users"] = self._base_table(columns=[ - ColumnDefinition(name="id", type="UInt64"), - ColumnDefinition(name="name", type="Nullable(String)"), - ]) + live.tables["users"] = self._base_table( + columns=[ + ColumnDefinition(name="id", type="UInt64"), + ColumnDefinition(name="name", type="Nullable(String)"), + ] + ) diffs = compare_schemas(local, live) modified = [d for d in diffs if d.status == DiffStatus.MODIFIED] @@ -193,8 +205,12 @@ def test_detects_partition_by_change(self): def test_views_fallback_to_raw_ddl(self): local = Schema(database="db") live = Schema(database="db") - local.views["v"] = ViewDefinition(name="v", select_query="SELECT 1", raw_ddl="CREATE VIEW v AS SELECT 1") - live.views["v"] = ViewDefinition(name="v", select_query="SELECT 2", raw_ddl="CREATE VIEW v AS SELECT 2") + local.views["v"] = ViewDefinition( + name="v", select_query="SELECT 1", raw_ddl="CREATE VIEW v AS SELECT 1" + ) + live.views["v"] = ViewDefinition( + name="v", select_query="SELECT 2", raw_ddl="CREATE VIEW v AS SELECT 2" + ) diffs = compare_schemas(local, live) modified = [d for d in diffs if d.status == DiffStatus.MODIFIED] @@ -218,7 +234,9 @@ def test_mixed_object_types(self): local.tables["t1"] = self._base_table("t1") live.tables["t1"] = self._base_table("t1") - local.views["v1"] = ViewDefinition(name="v1", select_query="SELECT 1", raw_ddl="CREATE VIEW v1 AS SELECT 1") + local.views["v1"] = ViewDefinition( + name="v1", select_query="SELECT 1", raw_ddl="CREATE VIEW v1 AS SELECT 1" + ) # v1 is local_only (not in live) live.dictionaries["d1"] = DictDefinition(name="d1", raw_ddl="CREATE DICTIONARY d1 (...)") @@ -239,13 +257,15 @@ def test_mixed_object_types(self): @pytest.fixture def diff_runner(tmp_path: Path, monkeypatch): monkeypatch.chdir(tmp_path) - (tmp_path / "config.yaml").write_text(""" + (tmp_path / "config.yaml").write_text( + """ environments: dev: host: localhost database: testdb user: test -""") +""" + ) monkeypatch.setenv("CH_DEV_PASSWORD", "pass") # Create a snapshot @@ -277,6 +297,7 @@ def test_in_sync_exits_0(self, mock_schema, mock_client, diff_runner, tmp_path): mock_schema.return_value = live from ch_migrate.cli import main + result = diff_runner.invoke(main, ["diff", "dev"]) assert result.exit_code == 0 @@ -298,6 +319,7 @@ def test_drift_exits_1(self, mock_schema, mock_client, diff_runner, tmp_path): mock_schema.return_value = live from ch_migrate.cli import main + result = diff_runner.invoke(main, ["diff", "dev"]) assert result.exit_code == 1 @@ -305,16 +327,19 @@ def test_drift_exits_1(self, mock_schema, mock_client, diff_runner, tmp_path): @patch("ch_migrate.introspect.get_live_schema") def test_no_snapshot_exits_1(self, mock_schema, mock_client, tmp_path, monkeypatch): monkeypatch.chdir(tmp_path) - (tmp_path / "config.yaml").write_text(""" + (tmp_path / "config.yaml").write_text( + """ environments: dev: host: localhost database: testdb user: test -""") +""" + ) monkeypatch.setenv("CH_DEV_PASSWORD", "pass") from ch_migrate.cli import main + runner = CliRunner() result = runner.invoke(main, ["diff", "dev"]) assert result.exit_code == 1 @@ -342,9 +367,14 @@ def test_render_with_drift(self): console = Console(file=output, force_terminal=True, width=80) diffs = [ - SchemaDiff(name="users", obj_type="table", status=DiffStatus.MODIFIED, field_diffs=[ - FieldDiff("engine", "MergeTree", "ReplacingMergeTree", "engine differs"), - ]), + SchemaDiff( + name="users", + obj_type="table", + status=DiffStatus.MODIFIED, + field_diffs=[ + FieldDiff("engine", "MergeTree", "ReplacingMergeTree", "engine differs"), + ], + ), SchemaDiff(name="new_table", obj_type="table", status=DiffStatus.REMOTE_ONLY), SchemaDiff(name="old_view", obj_type="view", status=DiffStatus.LOCAL_ONLY), ] diff --git a/tests/test_hooks.py b/tests/test_hooks.py index b4e1b61..9abc30a 100644 --- a/tests/test_hooks.py +++ b/tests/test_hooks.py @@ -3,7 +3,9 @@ from __future__ import annotations from pathlib import Path -from unittest.mock import MagicMock +from unittest.mock import MagicMock, call + +import pytest from ch_migrate.hooks import HookRegistry, run_hooks @@ -227,7 +229,6 @@ class TestUpgradeEnvCommand: def test_upgrade_env_creates_new_env_py(self, tmp_path: Path): """upgrade-env copies the package env.py when no existing env.py.""" from click.testing import CliRunner - from ch_migrate.cli import main migrations_dir = tmp_path / "migrations" @@ -244,7 +245,6 @@ def test_upgrade_env_creates_new_env_py(self, tmp_path: Path): def test_upgrade_env_backs_up_existing(self, tmp_path: Path, monkeypatch): """upgrade-env creates a .bak backup of existing env.py.""" from click.testing import CliRunner - from ch_migrate.cli import main migrations_dir = tmp_path / "migrations" @@ -268,7 +268,6 @@ def test_upgrade_env_backs_up_existing(self, tmp_path: Path, monkeypatch): def test_upgrade_env_no_migrations_dir(self, tmp_path: Path, monkeypatch): """upgrade-env fails gracefully when no migrations dir exists.""" from click.testing import CliRunner - from ch_migrate.cli import main monkeypatch.chdir(tmp_path) diff --git a/tests/test_json_output.py b/tests/test_json_output.py deleted file mode 100644 index 2aabdc3..0000000 --- a/tests/test_json_output.py +++ /dev/null @@ -1,196 +0,0 @@ -"""CLI JSON contracts: graph meaning, error exits, waivers, and structural drift.""" - -import json -from pathlib import Path -from unittest.mock import MagicMock - -import pytest -from click.testing import CliRunner -from jsonschema import Draft202012Validator - -from ch_migrate.cli import main -from ch_migrate.connection import MigrationState -from ch_migrate.introspect import Schema, parse_create_statement -from ch_migrate.version_table import VersionTableState - -SCHEMAS = Path(__file__).parents[1] / "docs" / "schemas" - - -@pytest.fixture -def json_project(tmp_path, monkeypatch): - runner = CliRunner() - assert runner.invoke(main, ["init", str(tmp_path)]).exit_code == 0 - monkeypatch.chdir(tmp_path) - monkeypatch.setattr("ch_migrate.cli.get_env_config", lambda *args: {"database": "example"}) - state = {"heads": set(), "error": None} - - def read_state(config): - if state["error"]: - raise OSError(state["error"]) - return MigrationState(state["heads"], VersionTableState("example", "Atomic")) - - monkeypatch.setattr("ch_migrate.connection.get_migration_state", read_state) - return tmp_path, state - - -@pytest.fixture -def revision_project(json_project): - root, state = json_project - versions = root / "migrations" / "versions" - _revision(versions / "root.py", None, 'irreversible = "Removed historical rows"') - _revision(versions / "left.py", "root") - _revision(versions / "right.py", "root") - _revision(versions / "merged.py", ("left", "right")) - return root, state - - -@pytest.mark.parametrize( - "heads,applied,pending,exit_code", - [ - (set(), [], ["left", "merged", "right", "root"], 1), - ({"merged"}, ["left", "merged", "right", "root"], [], 0), - ({"right"}, ["right", "root"], ["left", "merged"], 1), - ({"foreign"}, [], ["left", "merged", "right", "root"], 1), - ({"root", "merged"}, ["left", "merged", "right", "root"], [], 1), - ], -) -def test_status_json_resolves_graph(revision_project, heads, applied, pending, exit_code): - # Pytest's parameterization is the public test contract, not a runtime API. - revision_project[1]["heads"] = heads - result, document = _run("status", "dev", "--json") - assert result.exit_code == exit_code, result.output - assert document["current_heads"] == sorted(heads) - assert document["script_heads"] == ["merged"] - assert document["applied"] == applied - assert document["pending"] == pending - assert document["at_head"] is (exit_code == 0) - assert document["database"] == "example" - - -def test_history_json_preserves_merge_and_irreversibility(revision_project): - revision_project[1]["heads"] = {"merged"} - result, document = _run("history", "dev", "--json") - assert result.exit_code == 0 - revisions = {item["revision"]: item for item in document["revisions"]} - assert revisions["merged"]["down_revisions"] == ["left", "right"] - assert revisions["root"]["irreversible"] == "Removed historical rows" - assert revisions["left"]["irreversible"] is None - assert all(item["applied"] is True for item in revisions.values()) - assert revisions["root"]["description"] == "Migration root" - assert revisions["root"]["create_date"] == "2026-10-02 12:00:00" - assert Path(revisions["root"]["path"]).name == "root.py" - - -@pytest.mark.parametrize("command", ["status", "history"]) -def test_revision_json_database_error_is_not_empty_success(revision_project, command): - revision_project[1]["error"] = "database unavailable" - result, document = _run(command, "dev", "--json") - assert result.exit_code == 2 - assert "database unavailable" in document["error"] - if command == "history": - assert all(item["applied"] is None for item in document["revisions"]) - else: - assert "at_head" not in document - human = CliRunner().invoke(main, [command, "dev"]) - assert human.exit_code == 0 - - -def test_history_json_unknown_head_does_not_claim_unapplied(revision_project): - revision_project[1]["heads"] = {"foreign"} - result, document = _run("history", "dev", "--json") - assert result.exit_code == 0 - assert all(item["applied"] is None for item in document["revisions"]) - assert "foreign" in result.stderr - - -@pytest.mark.parametrize("command", ["status", "history", "lint", "diff"]) -def test_json_config_errors_are_documents(json_project, monkeypatch, command): - def fail_config(*args): - raise ValueError("environment is not configured") - - monkeypatch.setattr("ch_migrate.cli.get_env_config", fail_config) - result, document = _run(command, "missing", "--json") - assert result.exit_code == (1 if command == "lint" else 2) - assert "environment is not configured" in document["error"] - - -def test_lint_json_reports_gate_and_waiver_reason(json_project): - root, _ = json_project - sql = root / "migrations" / "sql" / "sample.up.sql" - sql.write_text( - "CREATE TABLE unsafe (id UInt64) ENGINE = MergeTree ORDER BY id;\n" - "-- ch-migrate: allow-non-idempotent Copy is manually reconciled before retry\n" - "INSERT INTO unsafe SELECT 1;\n" - ) - _revision( - root / "migrations" / "versions" / "sample.py", - None, - "from ch_migrate import run_sql\ndef upgrade():\n run_sql('sample.up.sql')\n", - ) - result, document = _run("lint", "--json") - assert result.exit_code == 1 - gate = [item for item in document["findings"] if item["rule"] == "idempotency"] - assert [(item["blocking"], item["waived"]) for item in gate] == [ - (True, None), - (False, "Copy is manually reconciled before retry"), - ] - assert [item["line"] for item in gate] == [1, 3] - assert all(item["file"].endswith("sample.up.sql") for item in gate) - assert document["counts"]["blocking"] == 1 - assert document["counts"]["waived"] == 1 - assert document["counts"]["error"] == 1 - assert CliRunner().invoke(main, ["lint"]).exit_code == result.exit_code - - -@pytest.mark.parametrize("drift", [False, True]) -def test_diff_json_reports_structural_changes(json_project, monkeypatch, drift): - root, _ = json_project - snapshot = root / "snapshot" / "tables" - snapshot.mkdir(parents=True) - before = "CREATE TABLE example.events (id UInt64) ENGINE = MergeTree ORDER BY id" - (snapshot / "events.sql").write_text(before) - live = Schema(database="example") - after = before.replace("id UInt64", "id UInt64, extra String") if drift else before - live.tables["events"] = parse_create_statement(after) - client = MagicMock() - monkeypatch.setattr("ch_migrate.connection.get_client", lambda config: client) - monkeypatch.setattr("ch_migrate.introspect.get_live_schema", lambda *args: live) - result, document = _run("diff", "dev", "--json", "--snapshot-dir", str(snapshot.parent)) - assert result.exit_code == int(drift), result.output - assert document["in_sync"] is (not drift) - item = document["objects"][0] - assert item["name"] == "events" and item["type"] == "table" - assert item["status"] == ("modified" if drift else "in_sync") - if drift: - assert any("extra" in detail["field"] for detail in item["details"]) - else: - assert item["details"] == [] - - -def test_diff_json_no_snapshot_returns_error(json_project): - result, document = _run("diff", "dev", "--json") - assert result.exit_code == 2 - assert "snapshot" in document["error"].lower() - assert "in_sync" not in document - - -def test_json_invalid_option_value_is_parseable(json_project): - result, document = _run("diff", "dev", "--json", "--snapshot-dir", "missing") - assert result.exit_code == 2 - assert "missing" in document["error"] - - -def _revision(path, parents, body=""): - path.write_text( - f'"""Migration {path.stem}\n\nCreate Date: 2026-10-02 12:00:00\n"""\n' - f"revision = {path.stem!r}\ndown_revision = {parents!r}\n{body}\n" - ) - - -def _run(*args): - result = CliRunner().invoke(main, list(args)) - document = json.loads(result.stdout) - schema = json.loads((SCHEMAS / f"{args[0]}.schema.json").read_text()) - Draft202012Validator.check_schema(schema) - Draft202012Validator(schema).validate(document) - return result, document diff --git a/tests/test_lint.py b/tests/test_lint.py index ca13666..5797e3a 100644 --- a/tests/test_lint.py +++ b/tests/test_lint.py @@ -14,6 +14,7 @@ STATIC_RULES, DestructiveChangeRule, IdempotencyRule, + LargeTableMutationRule, LintConfig, LintReport, LintResult, @@ -24,16 +25,27 @@ lint_migrations, ) + # --------------------------------------------------------------------------- # LintConfig tests # --------------------------------------------------------------------------- class TestLintConfig: + def test_defaults(self): + config = LintConfig() + assert config.large_table_threshold == 100_000_000 + assert config.rules == {} + + def test_from_config_empty(self): + config = LintConfig.from_config({}) + assert config.large_table_threshold == 100_000_000 + def test_from_config_with_values(self): config = LintConfig.from_config( { "lint": { + "large_table_threshold": 50_000, "rules": { "destructive_changes": "error", "missing_on_cluster": "off", @@ -41,6 +53,7 @@ def test_from_config_with_values(self): } } ) + assert config.large_table_threshold == 50_000 assert config.rules["destructive_changes"] == Severity.ERROR assert config.rules["missing_on_cluster"] == Severity.OFF @@ -54,20 +67,6 @@ def test_from_config_invalid_severity_ignored(self): ) assert "destructive_changes" not in config.rules - def test_retired_size_config_is_ignored_with_one_warning(self, capsys): - config = LintConfig.from_config( - { - "lint": { - "large_table_threshold": 1, - "rules": {"large_table_mutation": "error", "destructive_changes": "warn"}, - } - } - ) - assert config.rules == {"destructive_changes": Severity.WARN} - warning = capsys.readouterr().err - assert len(warning.splitlines()) == 1 - assert "ch-migrate plan" in warning - # --------------------------------------------------------------------------- # LintReport tests @@ -274,6 +273,45 @@ def test_flags_alter_and_drop(self): assert len(results) == 2 +# --------------------------------------------------------------------------- +# LargeTableMutationRule tests +# --------------------------------------------------------------------------- + + +class TestLargeTableMutationRule: + def _make_client(self, row_count: int) -> MagicMock: + client = MagicMock() + result = MagicMock() + result.result_rows = [[row_count]] + client.query.return_value = result + return client + + def test_flags_large_table(self): + client = self._make_client(200_000_000) + sql = "ALTER TABLE mydb.users ADD COLUMN phone String" + results = LargeTableMutationRule().check(sql, client=client, database="mydb") + assert len(results) == 1 + assert "200,000,000" in results[0].message + + def test_passes_small_table(self): + client = self._make_client(1000) + sql = "ALTER TABLE mydb.users ADD COLUMN phone String" + results = LargeTableMutationRule().check(sql, client=client, database="mydb") + assert results == [] + + def test_respects_custom_threshold(self): + client = self._make_client(5000) + config = LintConfig(large_table_threshold=1000) + sql = "ALTER TABLE mydb.users ADD COLUMN phone String" + results = LargeTableMutationRule().check(sql, client=client, database="mydb", config=config) + assert len(results) == 1 + + def test_skips_without_client(self): + sql = "ALTER TABLE mydb.users ADD COLUMN phone String" + results = LargeTableMutationRule().check(sql) + assert results == [] + + # --------------------------------------------------------------------------- # MVDependencyRule tests # --------------------------------------------------------------------------- @@ -283,9 +321,9 @@ class TestMVDependencyRule: def _make_client_with_deps(self) -> MagicMock: """Create a mock client that returns a dependency graph with MV on events.""" from ch_migrate.introspect import ( + DepType, DependencyEdge, DependencyGraph, - DepType, ObjectNode, ) diff --git a/tests/test_mv_validate.py b/tests/test_mv_validate.py index 70cf803..ba97eba 100644 --- a/tests/test_mv_validate.py +++ b/tests/test_mv_validate.py @@ -5,7 +5,11 @@ import textwrap from pathlib import Path +import pytest + from ch_migrate.mv_validate import ( + MVDeclaration, + MVValidationError, _find_grant_inserts, _find_grant_selects, _find_permissive_row_policies, @@ -15,6 +19,7 @@ validate_mv_migrations, ) + # --------------------------------------------------------------------------- # Helpers # --------------------------------------------------------------------------- @@ -861,7 +866,6 @@ class TestCLIIntegration: def test_up_does_not_block_on_mv_warnings(self, tmp_path: Path, monkeypatch): from click.testing import CliRunner - from ch_migrate.cli import main # Create minimal project structure @@ -901,7 +905,6 @@ def test_up_does_not_block_on_mv_warnings(self, tmp_path: Path, monkeypatch): def test_up_skip_mv_check_bypasses(self, tmp_path: Path, monkeypatch): from click.testing import CliRunner - from ch_migrate.cli import main (tmp_path / "config.yaml").write_text( @@ -939,7 +942,7 @@ class TestLintIntegration: """Test MVDeclarationRule via lint_migrations.""" def test_lint_catches_missing_declarations(self, tmp_path: Path): - from ch_migrate.lint import Severity, lint_migrations + from ch_migrate.lint import LintConfig, Severity, lint_migrations versions = tmp_path / "versions" _write_migration( diff --git a/tests/test_new.py b/tests/test_new.py index 7590a91..91b91c8 100644 --- a/tests/test_new.py +++ b/tests/test_new.py @@ -7,7 +7,7 @@ from click.testing import CliRunner from ch_migrate import IrreversibleMigration -from ch_migrate.authoring import NewOptions, SqlFiles, read_revision_header, render_revision +from ch_migrate.authoring import SqlFiles, read_revision_header, render_revision from ch_migrate.cli import main from ch_migrate.lint import lint_migrations from ch_migrate.mv_validate import validate_mv_migrations @@ -146,86 +146,11 @@ def test_rewrite_preserves_header_and_merge_metadata(tmp_path): "branch_labels = ('topic',)\ndepends_on = ('dependency',)\n" ) path.write_text(original) - result = render_revision( - read_revision_header(path), SqlFiles("up.sql", "down.sql"), NewOptions() - ) + result = render_revision(read_revision_header(path), SqlFiles("up.sql", "down.sql"), None) assert ast.get_docstring(ast.parse(result)) == ast.get_docstring(ast.parse(original)) assert _assignments(result) == _assignments(original) -def test_new_rebuild_fetches_portable_ddl_and_marks_irreversible(root, monkeypatch): - from ch_migrate.sql import split_statements - from ch_migrate.statements import migration_statements - - ddl = ( - "CREATE TABLE `demo_dev`.`logs` UUID 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa' " - "ON CLUSTER `dev_cluster` (id UInt64, value String DEFAULT 'demo_dev.logs') " - "ENGINE = MergeTree ORDER BY id" - ) - monkeypatch.setattr("ch_migrate.scaffold.fetch_current_ddl", lambda *_: ddl) - _new("reorder", ["--table", "logs", "--rebuild"]) - [revision] = list((root / "migrations" / "versions").glob("*.py")) - [upgrade] = list((root / "migrations" / "sql").rglob("*.up.sql")) - assert upgrade.parent == root / "migrations" / "sql" / "history" / "tables" / "logs" - assert list((root / "migrations" / "sql").rglob("*.down.sql")) == [] - sql = split_statements(upgrade.read_text())[0].sql - assert sql.startswith("CREATE TABLE {db}.`logs`") - assert "UUID" not in sql and "ON CLUSTER" not in sql - assert "DEFAULT 'demo_dev.logs'" in sql - statement = migration_statements(revision)[0] - assert statement.rebuild.table == "logs" - assert statement.rebuild.create_sql_path == str( - upgrade.relative_to(root / "migrations" / "sql") - ) - namespace = {} - exec(compile(revision.read_text(), str(revision), "exec"), namespace) - with pytest.raises(IrreversibleMigration): - namespace["downgrade"]() - - -def test_new_rebuild_offline_placeholder_cannot_execute(root, monkeypatch): - from ch_migrate.sql import split_statements - from ch_migrate.statements import migration_statements - - monkeypatch.setattr("ch_migrate.scaffold.fetch_current_ddl", lambda *_: None) - result = CliRunner().invoke(main, ["new", "dev", "offline", "--table", "logs", "--rebuild"]) - assert result.exit_code == 0, result.output - [upgrade] = list((root / "migrations" / "sql").rglob("*.up.sql")) - assert split_statements(upgrade.read_text()) == [] - assert "SHOW CREATE TABLE" in upgrade.read_text() - [revision] = list((root / "migrations" / "versions").glob("*.py")) - with pytest.raises(ValueError, match="exactly one"): - migration_statements(revision) - assert _assignments(revision.read_text())["irreversible"] - - -@pytest.mark.parametrize( - "options", - [ - ["--rebuild"], - ["--rebuild", "--view", "logs"], - ["--rebuild", "--table", "logs", "--exchange"], - ["--rebuild", "--table", "logs", "--python"], - ["--rebuild", "--table", "logs", "--irreversible", "reason"], - ], -) -def test_new_rebuild_conflicts_leave_no_artifacts(root, options): - result = CliRunner().invoke(main, ["new", "dev", "invalid", *options]) - assert result.exit_code != 0 - assert list((root / "migrations" / "versions").glob("*.py")) == [] - assert list((root / "migrations" / "sql").rglob("*.sql")) == [] - - -def test_new_rebuild_replacement_keeps_exchange_deprecation(root, monkeypatch): - monkeypatch.setattr("ch_migrate.scaffold.fetch_current_ddl", lambda *_: None) - monkeypatch.setattr("ch_migrate.scaffold.find_dependent_dictionaries", lambda *_: []) - result = CliRunner().invoke(main, ["new", "dev", "legacy", "--table", "logs", "--exchange"]) - assert result.exit_code == 0, result.output - assert "Deprecated" in result.stderr and "--rebuild" in result.stderr - assert "lose rows" in result.stderr - assert len(list((root / "migrations" / "versions").glob("*.py"))) == 1 - - def _new(name, options): result = CliRunner().invoke(main, ["new", "dev", name, *options]) assert result.exit_code == 0, result.output diff --git a/tests/test_plan.py b/tests/test_plan.py deleted file mode 100644 index 9b42b7b..0000000 --- a/tests/test_plan.py +++ /dev/null @@ -1,73 +0,0 @@ -"""Plan JSON error behavior and schema's consumer-visible constraints.""" - -import json -from pathlib import Path - -import pytest -from click.testing import CliRunner -from jsonschema import Draft202012Validator, ValidationError - -from ch_migrate.cli import main - -SCHEMA_PATH = Path(__file__).parents[1] / "docs/schemas/plan.schema.json" - - -@pytest.fixture -def plan_project(tmp_path, monkeypatch): - initialized = CliRunner().invoke(main, ["init", str(tmp_path)]) - assert initialized.exit_code == 0, initialized.output - monkeypatch.chdir(tmp_path) - return tmp_path - - -@pytest.fixture -def validator(): - schema = json.loads(SCHEMA_PATH.read_text()) - Draft202012Validator.check_schema(schema) - return Draft202012Validator(schema) - - -@pytest.mark.parametrize("environment", ["missing", "dev"]) -def test_plan_configuration_failure_is_a_schema_valid_error(plan_project, validator, environment): - result = CliRunner().invoke(main, ["plan", environment, "--json"]) - assert result.exit_code == 2, result.output - document = json.loads(result.stdout) - validator.validate(document) - assert document["schema_version"] == 1 - assert document["command"] == "plan" - assert isinstance(document["error"], str) and document["error"] - assert "migrations" not in document - - -def test_plan_schema_rejects_false_precision_and_missing_migration_fields(validator): - document = { - "schema_version": 1, - "command": "plan", - "database": "example", - "environment": "dev", - "gate_would_refuse": False, - "warnings": [], - "findings": [], - "counts": { - "total": 0, - "error": 0, - "warning": 0, - "info": 0, - "blocking": 0, - "waived": 0, - }, - "migrations": [], - } - validator.validate(document) - with pytest.raises(ValidationError): - validator.validate({**document, "gate_would_refuse": "false"}) - with pytest.raises(ValidationError): - validator.validate({**document, "migrations": [{"revision": "aaaa"}]}) - with pytest.raises(ValidationError): - validator.validate({**document, "counts": {**document["counts"], "blocking": -1}}) - - -def test_plan_unknown_environment_does_not_suppress_error_with_empty_success(plan_project): - result = CliRunner().invoke(main, ["plan", "not-configured"]) - assert result.exit_code != 0 - assert "not-configured" in result.output or "environment" in result.output.lower() diff --git a/tests/test_rebuild_authoring.py b/tests/test_rebuild_authoring.py deleted file mode 100644 index 5cc242e..0000000 --- a/tests/test_rebuild_authoring.py +++ /dev/null @@ -1,98 +0,0 @@ -"""Guarded rebuild extraction and the migration gate's public behavior.""" - -from pathlib import Path - -import pytest - -from ch_migrate.classify import classify -from ch_migrate.lint import GATE_RULES, lint_migrations -from ch_migrate.statements import migration_statements - - -@pytest.fixture -def rebuild_project(tmp_path): - versions = tmp_path / "migrations" / "versions" - versions.mkdir(parents=True) - sql = tmp_path / "migrations" / "sql" - sql.mkdir() - (sql / "replacement.sql").write_text( - "CREATE TABLE {db}.events (id UInt64, value UInt64) ENGINE = MergeTree ORDER BY value;\n" - ) - return versions - - -def test_rebuild_keeps_operation_order_sql_origin_and_options(rebuild_project): - revision = _revision( - rebuild_project, - """ -def upgrade(): - op.execute('SELECT 1') - op.rebuild_table('events', 'replacement.sql', select='id, value', allow_unacknowledged_async_loss=True) - op.execute('SELECT 2') -""", - ) - statements = migration_statements(revision) - assert statements[0].sql == "SELECT 1" - assert statements[2].sql == "SELECT 2" - rebuild = statements[1] - assert rebuild.source == "migrations/sql/replacement.sql" - assert rebuild.line == 1 - assert rebuild.rebuild.table == "events" - assert rebuild.rebuild.select == "id, value" - assert rebuild.rebuild.allow_unacknowledged_async_loss is True - assert classify(rebuild).kind == "rebuild" - assert classify(rebuild).table == "events" - - -def test_only_guarded_rebuild_is_exempt_from_create_idempotency(rebuild_project): - _revision( - rebuild_project, - """ -def upgrade(): - op.rebuild_table('events', 'replacement.sql') - op.execute('CREATE TABLE unsafe (id UInt64) ENGINE = Memory') -""", - ) - report = lint_migrations(rebuild_project) - blockers = [item for item in report.results if item.rule in GATE_RULES] - assert len(blockers) == 1 - assert blockers[0].file == "migrations/versions/a.py" - assert "CREATE TABLE unsafe" in blockers[0].statement - - -def test_exported_helper_keyword_arguments_are_extracted(rebuild_project): - revision = _revision( - rebuild_project, - """ -def upgrade(): - rebuild_table(table=f'{db}.events', create_sql_path='replacement.sql') -""", - ) - statement = migration_statements(revision)[0] - assert statement.rebuild.table == "{db}.events" - assert statement.rebuild.allow_unacknowledged_async_loss is False - - -@pytest.mark.parametrize("argument", ["unknown_path", "choose_path()"]) -def test_unreadable_rebuild_definition_is_not_silently_omitted(rebuild_project, argument): - revision = _revision( - rebuild_project, f"def upgrade():\n op.rebuild_table('events', {argument})\n" - ) - with pytest.raises(ValueError, match="statically readable"): - migration_statements(revision) - - -def test_rebuild_rejects_multiple_statements_before_gate_exemption(rebuild_project): - path = rebuild_project.parent / "sql" / "replacement.sql" - path.write_text(path.read_text() + "DROP TABLE another;\n") - revision = _revision( - rebuild_project, "def upgrade():\n op.rebuild_table('events', 'replacement.sql')\n" - ) - with pytest.raises(ValueError, match="exactly one"): - migration_statements(revision) - - -def _revision(versions: Path, body: str) -> Path: - path = versions / "a.py" - path.write_text("revision = 'a'\ndown_revision = None\n" + body) - return path diff --git a/tests/test_rebuild_ddl.py b/tests/test_rebuild_ddl.py deleted file mode 100644 index 785eafa..0000000 --- a/tests/test_rebuild_ddl.py +++ /dev/null @@ -1,290 +0,0 @@ -"""Semantic tests of rebuild helper schemas and rejection boundaries.""" - -from __future__ import annotations - -import pytest - -from ch_migrate.introspect import TableDefinition -from ch_migrate.rebuild_ddl import ( - _parse, - build_definition, - dual_ddl, - helper_ddl, - target_comment, -) -from ch_migrate.rebuild_types import HelperIdentity, RebuildOptions - -SOURCE = """CREATE TABLE `analytics`.`events` UUID 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa' ( - `id` UInt64 CODEC(Delta, ZSTD(3)), - `day` Date DEFAULT toDate('2026-10-04'), - `kind` LowCardinality(String), - `computed` UInt64 MATERIALIZED id * 2, - `virtual` String ALIAS toString(id), - INDEX ix_kind kind TYPE bloom_filter(0.01) GRANULARITY 2, - PROJECTION by_day (SELECT day, count() GROUP BY day) -) ENGINE = ReplicatedReplacingMergeTree('/clickhouse/tables/{shard}/analytics/events', '{replica}', id) -PARTITION BY toYYYYMM(day) PRIMARY KEY id ORDER BY (id, day) -TTL day + INTERVAL 90 DAY SETTINGS index_granularity = 4096, allow_nullable_key = 0""" -TARGET = """CREATE TABLE analytics.events ( - `id` UInt64 CODEC(Delta, ZSTD(3)), - `day` Date DEFAULT toDate('2026-10-04'), - `kind` LowCardinality(String), - `computed` UInt64 MATERIALIZED id * 2, - `virtual` String ALIAS toString(id), - INDEX ix_kind kind TYPE bloom_filter(0.01) GRANULARITY 2, - PROJECTION by_day (SELECT day, count() GROUP BY day) -) ENGINE = ReplicatedReplacingMergeTree('/clickhouse/tables/{shard}/analytics/events', '{replica}', id) -PARTITION BY toYYYYMM(day) PRIMARY KEY id ORDER BY (day, id) -TTL day + INTERVAL 90 DAY SETTINGS index_granularity = 4096, allow_nullable_key = 0""" -UUIDS = { - "snap": "bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb", - "new": "cccccccc-cccc-cccc-cccc-cccccccccccc", - "stage": "dddddddd-dddd-dddd-dddd-dddddddddddd", -} - - -def _definition(target: str = TARGET, source: str = SOURCE, **overrides): - options = RebuildOptions("analytics", "events", "r1", 3, **overrides) - return build_definition( - TableDefinition("events", "ReplicatedReplacingMergeTree", raw_ddl=source), target, options - ) - - -def test_helpers_preserve_complete_physical_schema_and_isolate_keeper_paths(): - definition = _definition(cluster="all_replicas") - paths = set() - for role, uuid in UUIDS.items(): - sql = helper_ddl(definition, role, HelperIdentity(uuid=uuid)) - parsed = _parse(sql, "analytics", "events__chm_" + role) - assert parsed.uuid_span is not None - assert sql[parsed.uuid_span[0] : parsed.uuid_span[1]] == f"UUID '{uuid}'" - assert parsed.cluster_span is not None - assert parsed.clauses["PARTITION"] == "toYYYYMM(day)" - assert parsed.clauses["PRIMARY"] == "id" - assert parsed.clauses["SETTINGS"] == "index_granularity = 4096, allow_nullable_key = 0" - assert "CODEC(Delta, ZSTD(3))" in sql - assert "INDEX ix_kind" in sql - assert "PROJECTION by_day" in sql - assert "CREATE TABLE" in sql and " AS analytics.events" not in sql - assert parsed.clauses["ORDER"] == ("(id, day)" if role == "snap" else "(day, id)") - paths.add(parsed.engine_args[0]) - assert len(paths) == 3 - assert all("/__chm_" in path for path in paths) - assert definition.columns == ("id", "day", "kind") - assert definition.target.order_by == ["day", "id"] - - -def test_snapshot_and_stage_are_plain_and_ttl_free_so_merges_keep_every_row(): - source = SOURCE.replace( - "`kind` LowCardinality(String),", "`kind` LowCardinality(String) TTL day + INTERVAL 1 DAY," - ) - definition = _definition( - TARGET.replace( - "`kind` LowCardinality(String),", - "`kind` LowCardinality(String) TTL day + INTERVAL 1 DAY CODEC(ZSTD(1)),", - ), - source, - ) - for role, uuid in UUIDS.items(): - parsed = _parse( - helper_ddl(definition, role, HelperIdentity(uuid=uuid)), - "analytics", - "events__chm_" + role, - ) - kind = next(column for column in parsed.columns if column.name == "kind") - if role == "new": - assert parsed.engine_name == "ReplicatedReplacingMergeTree" - assert parsed.engine_args[1:] == ("'{replica}'", "id") - assert parsed.clauses["TTL"] == "day + INTERVAL 90 DAY" - assert "TTL" in parsed.sql[parsed.schema_at : parsed.schema_end] - continue - assert parsed.engine_name == "ReplicatedMergeTree" - assert parsed.engine_args[1:] == ("'{replica}'",) - assert "TTL" not in parsed.clauses - assert "TTL" not in parsed.sql[parsed.schema_at : parsed.schema_end] - assert kind.type == "LowCardinality(String)" - assert kind.codec == ("(ZSTD(1))" if role == "stage" else None) - assert parsed.clauses["SETTINGS"] == "index_granularity = 4096, allow_nullable_key = 0" - - -def test_implicit_keeper_arguments_get_explicit_unique_paths_preserving_versions(): - source = SOURCE.replace( - "ReplicatedReplacingMergeTree('/clickhouse/tables/{shard}/analytics/events', '{replica}', id)", - "ReplicatedReplacingMergeTree(id)", - ) - target = TARGET.replace( - "ReplicatedReplacingMergeTree('/clickhouse/tables/{shard}/analytics/events', '{replica}', id)", - "ReplicatedReplacingMergeTree(id)", - ) - definition = _definition(target, source) - paths = [] - for role, uuid in UUIDS.items(): - parsed = _parse( - helper_ddl(definition, role, HelperIdentity(uuid=uuid)), - "analytics", - "events__chm_" + role, - ) - params = ("id",) if role == "new" else () - assert parsed.engine_args == ( - f"'/clickhouse/tables/{uuid}/{{shard}}'", - "'{replica}'", - *params, - ) - paths.append(parsed.engine_args[0]) - assert len(set(paths)) == 3 - - -def test_zero_arg_engine_and_nonreplicated_engine_do_not_share_keeper_paths(): - source = SOURCE.replace( - "ReplicatedReplacingMergeTree('/clickhouse/tables/{shard}/analytics/events', '{replica}', id)", - "ReplicatedMergeTree", - ) - definition = _definition( - TARGET.replace( - "ReplicatedReplacingMergeTree('/clickhouse/tables/{shard}/analytics/events', '{replica}', id)", - "ReplicatedMergeTree()", - ), - source, - ) - assert _parse( - helper_ddl(definition, "snap", HelperIdentity(uuid=UUIDS["snap"])), - "analytics", - "events__chm_snap", - ).engine_args == (f"'/clickhouse/tables/{UUIDS['snap']}/{{shard}}'", "'{replica}'") - plain = _definition( - TARGET.replace( - "ReplicatedReplacingMergeTree('/clickhouse/tables/{shard}/analytics/events', '{replica}', id)", - "MergeTree()", - ), - SOURCE.replace( - "ReplicatedReplacingMergeTree('/clickhouse/tables/{shard}/analytics/events', '{replica}', id)", - "MergeTree()", - ), - ) - assert ( - _parse( - helper_ddl(plain, "new", HelperIdentity(uuid=UUIDS["new"])), - "analytics", - "events__chm_new", - ).engine_args - == () - ) - - -def test_dual_projection_explicitly_names_insertable_target_columns(): - definition = _definition(select="id, toDate(day) AS day, concat(kind, ',', kind)") - sql = dual_ddl(definition, HelperIdentity(uuid=UUIDS["new"])) - assert "TO `analytics`.`events__chm_new` AS SELECT" in sql - assert "(id) AS `id`, (toDate(day)) AS `day`, (concat(kind, ',', kind)) AS `kind`" in sql - assert sql.endswith("FROM `analytics`.`events`") - assert "computed" not in sql and "virtual" not in sql - assert "SELECT `id`, `day`, `kind` FROM" in dual_ddl( - _definition(), HelperIdentity(uuid=UUIDS["new"]) - ) - - -@pytest.mark.parametrize( - "sql", - [ - TARGET.replace("analytics.events", "other.events", 1), - TARGET.replace("analytics.events", "analytics.other", 1), - TARGET.replace("(\n `id`", "AS source (\n `id`", 1), - TARGET + "; CREATE TABLE analytics.second (id UInt64) ENGINE = MergeTree ORDER BY id", - TARGET.replace("ENGINE =", "AS source ENGINE =", 1), - TARGET.replace("ENGINE =", "UUID 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa' ENGINE =", 1), - TARGET.replace("ENGINE =", "ENGINE = Distributed('cluster', db, table) SETTINGS x=1 --", 1), - ], -) -def test_ambiguous_or_wrong_target_is_refused(sql): - with pytest.raises(ValueError): - _definition(sql) - - -@pytest.mark.parametrize( - "projection", - [ - "SELECT id, day, kind FROM another", - "id, day FROM another, kind", - "id, day", - "id, day, kind; DROP TABLE analytics.events", - "id, day AS other, kind", - ], -) -def test_invalid_projection_is_refused(projection): - with pytest.raises(ValueError): - _definition(select=projection) - - -def test_unknown_role_and_invalid_uuid_are_refused(): - definition = _definition() - with pytest.raises(ValueError): - helper_ddl(definition, "old", HelperIdentity(uuid=UUIDS["snap"])) - with pytest.raises(ValueError): - helper_ddl(definition, "snap", HelperIdentity(uuid="not-a-uuid")) - - -def test_helper_keeper_roots_survive_source_cleanup_and_table_rename(): - source = SOURCE.replace("/analytics/events", "/{database}/{table}") - target = TARGET.replace("/analytics/events", "/{database}/{table}") - definition = _definition(target, source) - original_root = "/clickhouse/tables/{shard}/analytics/events" - for role, uuid in UUIDS.items(): - parsed = _parse( - helper_ddl(definition, role, HelperIdentity(uuid=uuid)), - "analytics", - "events__chm_" + role, - ) - path = parsed.engine_args[0].strip("'") - assert "{table}" not in path and "{database}" not in path - assert path != original_root - assert not path.startswith(original_root + "/") - - -# ClickHouse Cloud reports Shared engines with its own Keeper arguments. -CLOUD_SOURCE = """CREATE TABLE analytics.events -( - `id` UInt64, - `v` UInt64, - `day` Date TTL day + toIntervalYear(1) -) -ENGINE = SharedReplacingMergeTree('/clickhouse/tables/{uuid}/{shard}', '{replica}', v) -PARTITION BY toYYYYMM(day) -ORDER BY id -TTL day + toIntervalYear(2) -SETTINGS index_granularity = 8192 -COMMENT 'live events'""" - - -def _cloud_definition(): - target = CLOUD_SOURCE.replace("ORDER BY id", "ORDER BY (day, id)") - return _definition(target, CLOUD_SOURCE) - - -def test_cloud_helpers_carry_owner_token_instead_of_a_chosen_uuid(): - definition = _cloud_definition() - for role in ("new", "snap", "stage"): - token = f"chm-rebuild:r1:{role}:abc" - sql = helper_ddl(definition, role, HelperIdentity(token=token)) - parsed = _parse(sql, "analytics", "events__chm_" + role) - assert parsed.uuid_span is None and "UUID" not in sql - assert parsed.clauses["COMMENT"] == f"'{token}'" - assert "live events" not in sql - if role == "new": - assert parsed.engine_name == "ReplacingMergeTree" and parsed.engine_args == ("v",) - assert parsed.clauses["TTL"] == "day + toIntervalYear(2)" - else: - assert parsed.engine_name == "MergeTree" and parsed.engine_args == () - assert "TTL" not in sql - - -def test_cloud_dual_view_puts_owner_token_before_its_select(): - sql = dual_ddl(_cloud_definition(), HelperIdentity(token="chm-rebuild:r1:dual:abc")) - assert "UUID" not in sql - assert "TO `analytics`.`events__chm_new` COMMENT 'chm-rebuild:r1:dual:abc' AS SELECT" in sql - - -def test_target_comment_restores_the_replacements_own_comment(): - sql, value = target_comment(_cloud_definition()) - assert sql == "ALTER TABLE `analytics`.`events__chm_new` MODIFY COMMENT 'live events'" - assert value == "live events" - assert target_comment(_definition())[1] == "" diff --git a/tests/test_rebuild_dependencies.py b/tests/test_rebuild_dependencies.py deleted file mode 100644 index 46c8164..0000000 --- a/tests/test_rebuild_dependencies.py +++ /dev/null @@ -1,193 +0,0 @@ -"""Behavioral boundaries for dependent discovery, replacement and validation.""" - -from __future__ import annotations - -from dataclasses import dataclass -from types import SimpleNamespace - -import pytest - -from ch_migrate.introspect import parse_create_dictionary -from ch_migrate.rebuild_dependents import ( - DependentInventory, - DependentQuery, - DependentValidationError, - DictionaryReload, - _dictionary_query, - _rewrite, - _RewriteTarget, - inspect_dependents, - reload_dictionaries, - validate_dependents, -) - - -@dataclass -class _Result: - result_rows: tuple = () - column_names: tuple[str, ...] = () - column_types: tuple = () - - -class _Client: - def __init__(self, headers=None): - self.sql = [] - self.headers = headers or {} - - def query(self, sql, parameters=None): - self.sql.append(sql) - if sql.startswith("EXPLAIN"): - return _Result() - if sql.startswith("SELECT * FROM (SELECT * FROM ") and "__chm_new" in sql: - names = self.headers.get("target", {}) - return _Result(column_names=tuple(names), column_types=tuple(names.values())) - if sql.startswith("SELECT * FROM ("): - names = self.headers.get("source", {}) - return _Result(column_names=tuple(names), column_types=tuple(names.values())) - return _Result() - - def command(self, sql): - self.sql.append(sql) - - -def test_rewrite_changes_only_same_database_relation_not_literals_comments_or_alias(): - sql = ( - "SELECT 'FROM events', events.id FROM events AS events " - "JOIN `analytics`.`events` e ON e.id = events.id " - "JOIN other.events x ON x.id = e.id -- FROM events\n" - "WHERE events.id > 0 /* JOIN events */" - ) - rewritten, count = _rewrite(sql, _RewriteTarget("analytics", "events", "events__chm_new")) - assert count == 2 - assert "FROM `analytics`.`events__chm_new` AS events" in rewritten - assert "JOIN `analytics`.`events__chm_new` e" in rewritten - assert "JOIN other.events x" in rewritten - assert "'FROM events'" in rewritten - assert "events.id > 0 /* JOIN events */" in rewritten - - -def test_dictionary_uses_declared_key_and_attributes_and_excludes_remote_sources(): - ddl = ( - "CREATE DICTIONARY analytics.lookup (`id` UInt64, `label` String, " - "`weight` Float64) PRIMARY KEY id SOURCE(CLICKHOUSE(" - "HOST 'localhost' DB 'analytics' TABLE 'events')) " - "LIFETIME(MIN 0 MAX 0) LAYOUT(HASHED())" - ) - dictionary = parse_create_dictionary(ddl) - assert _dictionary_query(dictionary, "analytics", "events") == ( - "SELECT `id`, `label`, `weight` FROM `analytics`.`events`" - ) - assert ( - _dictionary_query( - parse_create_dictionary(ddl.replace("localhost", "remote")), "analytics", "events" - ) - is None - ) - assert ( - _dictionary_query( - parse_create_dictionary(ddl.replace("DB 'analytics'", "DB 'other'")), - "analytics", - "events", - ) - is None - ) - assert ( - _dictionary_query( - parse_create_dictionary(ddl.replace("CLICKHOUSE", "MYSQL")), "analytics", "events" - ) - is None - ) - - -def test_inspection_includes_source_mv_inner_engine_and_target_mv(monkeypatch): - schema = SimpleNamespace( - views={"v": SimpleNamespace(select_query="SELECT id FROM analytics.events")}, - materialized_views={ - "inner": SimpleNamespace(select_query="SELECT id FROM events", target_table=None), - "writer": SimpleNamespace( - select_query="SELECT id FROM other.source", target_table="events" - ), - "elsewhere": SimpleNamespace( - select_query="SELECT id FROM other.events", target_table="other.events" - ), - }, - dictionaries={}, - ) - monkeypatch.setattr("ch_migrate.rebuild_dependents.get_live_schema", lambda *_: schema) - client = _Client() - inventory = inspect_dependents(client, "analytics", "events") - assert [d.name for d in inventory.source_views] == ["v", "inner"] - assert [d.name for d in inventory.target_views] == ["writer"] - assert not inventory.dictionaries - - -def test_validation_refuses_missing_source_projection_with_named_error(): - definition = SimpleNamespace(database="analytics", table="events") - inventory = DependentInventory( - (DependentQuery("broken", "SELECT obsolete FROM events", "materialized_view"),), (), () - ) - - class RejectMissing(_Client): - def query(self, sql, parameters=None): - if sql.startswith("EXPLAIN"): - raise RuntimeError("Unknown identifier obsolete") - return super().query(sql, parameters) - - with pytest.raises( - DependentValidationError, match="materialized_view analytics.broken.*obsolete" - ): - validate_dependents(RejectMissing(), definition, inventory) - - -def test_target_view_names_a_server_rejected_conversion(): - class RejectCast(_Client): - def query(self, sql, parameters=None): - if sql.startswith("EXPLAIN PLAN SELECT CAST("): - raise RuntimeError("Cannot convert Array(UInt64) to UInt64") - return super().query(sql, parameters) - - client = RejectCast({"source": {"id": "Array(UInt64)"}, "target": {"id": "UInt64"}}) - definition = SimpleNamespace(database="analytics", table="events") - inventory = DependentInventory( - (), - (DependentQuery("writer", "SELECT [id] AS id FROM other.source", "materialized_view"),), - (), - ) - with pytest.raises(DependentValidationError, match="writer.*Cannot convert Array"): - validate_dependents(client, definition, inventory) - - -def test_dictionary_projection_rejects_missing_attribute(): - ddl = ( - "CREATE DICTIONARY analytics.lookup (id UInt64, label String) PRIMARY KEY id " - "SOURCE(CLICKHOUSE(DB 'analytics' TABLE 'events')) LAYOUT(HASHED())" - ) - - class DictionaryClient(_Client): - def query(self, sql, parameters=None): - if sql.startswith("SHOW CREATE"): - return _Result(result_rows=((ddl,),)) - return super().query(sql, parameters) - - client = DictionaryClient({"source": {"id": "UInt64"}}) - inventory = DependentInventory( - (), (), (DependentQuery("lookup", "SELECT id, label FROM events", "dictionary"),) - ) - with pytest.raises(DependentValidationError, match="lookup.*label"): - validate_dependents( - client, SimpleNamespace(database="analytics", table="events"), inventory - ) - - -def test_reload_uses_effective_cluster_and_names_failure(): - class FailingClient(_Client): - def command(self, sql): - super().command(sql) - if sql.endswith("`lookup2`"): - raise RuntimeError("reload failed") - - client = FailingClient() - with pytest.raises(DependentValidationError, match="analytics.lookup2.*reload failed"): - reload_dictionaries( - client, DictionaryReload("analytics", ("lookup", "lookup2"), "generated") - ) diff --git a/tests/test_rebuild_ownership.py b/tests/test_rebuild_ownership.py deleted file mode 100644 index b9ef9b0..0000000 --- a/tests/test_rebuild_ownership.py +++ /dev/null @@ -1,80 +0,0 @@ -"""Cloud rebuild ownership: an owner token proves an object is ours; nothing else is adopted.""" - -from __future__ import annotations - -from types import SimpleNamespace - -import pytest - -from ch_migrate.rebuild_engine import _Context -from ch_migrate.waiting_types import UnknownOutcome - -HOSTS = {"replica-a", "replica-b"} -TOKEN = "chm-rebuild:r1:snap:abc" - - -def test_unbound_helper_with_our_token_everywhere_binds_its_server_uuid(): - ctx = _cloud_context({host: ("u1", "MergeTree", TOKEN) for host in HOSTS}) - assert ctx.proven("snap") - assert ctx.bound_uuid("snap") == "u1" - assert ctx.state["helpers"]["snap"] == "u1" - assert ctx.saved == 1 - - -def test_absent_or_partly_visible_helper_is_not_proven_and_cannot_bind(): - ctx = _cloud_context({"replica-a": ("u1", "MergeTree", TOKEN)}) - assert not ctx.proven("snap") - with pytest.raises(UnknownOutcome): - ctx.bound_uuid("snap") - - -def test_another_owners_comment_is_never_adopted(): - catalog = {"replica-a": ("u1", "MergeTree", TOKEN), "replica-b": ("u1", "MergeTree", "x")} - ctx = _cloud_context(catalog) - with pytest.raises(UnknownOutcome, match="another owner"): - ctx.proven("snap") - - -def test_replicas_disagreeing_on_uuid_cannot_bind(): - catalog = {"replica-a": ("u1", "MergeTree", TOKEN), "replica-b": ("u2", "MergeTree", TOKEN)} - ctx = _cloud_context(catalog) - assert not ctx.proven("snap") - with pytest.raises(UnknownOutcome): - ctx.bound_uuid("snap") - - -def test_bound_helper_whose_comment_was_edited_fails_closed(): - ctx = _cloud_context({host: ("u1", "MergeTree", "edited") for host in HOSTS}) - ctx.state["helpers"]["snap"] = "u1" - with pytest.raises(UnknownOutcome, match="another owner"): - ctx.proven("snap") - - -def test_bound_new_is_proven_by_uuid_after_its_comment_is_restored(): - ctx = _cloud_context({host: ("u9", "ReplacingMergeTree", "live events") for host in HOSTS}) - ctx.state["helpers"]["new"] = "u9" - assert ctx.proven("new") - ctx.state["helpers"]["new"] = "other" - assert not ctx.proven("new") - - -def _cloud_context(catalog): - ctx = object.__new__(_Context) - ctx.shared = True - ctx.hosts = HOSTS - ctx.table = "events" - ctx.saved = 0 - record = { - "rebuild": { - "helpers": {"new": None, "snap": None, "stage": None}, - "tokens": {role: f"chm-rebuild:r1:{role}:abc" for role in ("new", "snap", "stage")}, - "dual_uuid": None, - } - } - - def checkpoint(): - ctx.saved += 1 - - ctx.runtime = SimpleNamespace(record=record, checkpoint=checkpoint) - ctx.catalog = lambda name: dict(catalog) - return ctx diff --git a/tests/test_rebuild_preflight.py b/tests/test_rebuild_preflight.py deleted file mode 100644 index 3b102d8..0000000 --- a/tests/test_rebuild_preflight.py +++ /dev/null @@ -1,303 +0,0 @@ -"""Behavioral checks for rebuild refusals and conservative writer inspection.""" - -from __future__ import annotations - -import pytest - -from ch_migrate.introspect import ColumnDefinition, TableDefinition -from ch_migrate.rebuild_preflight import RebuildRequest, inspect_rebuild - - -class _Result: - def __init__(self, rows): - self.result_rows = rows - - -class _Client: - def __init__(self, *, writer_settings=None, profile_value="0", mutation=False, shards=1): - self.writer_settings = ( - writer_settings if writer_settings is not None else {"async_insert": "1"} - ) - self.profile_value = profile_value - self.mutation = mutation - self.shards = shards - self.queries = [] - - def query(self, sql): - self.queries.append(sql) - if "uniqExact(shard_num)" in sql: - return _Result([(self.shards,)]) - if "name = 'part_log'" in sql: - return _Result([("node", 1)]) - if "engine = 'Distributed'" in sql: - return _Result([]) - if "system.mutations" in sql: - return _Result([("node", "mutation_1")] if self.mutation else []) - if "system.replicas" in sql: - return _Result([("node", "/group", "r1", 1)]) - if "system.parts" in sql: - return _Result([("node", "202601", "default", 9, 100)]) - if "system.disks" in sql: - return _Result([("node", "default", 1000)]) - if "system.merge_tree_settings" in sql: - return _Result([("node", "10")]) - if "system.asynchronous_inserts" in sql: - return _Result([]) - if "system.processes" in sql: - return _Result([]) - if "query_log" in sql and "sumIf" in sql: - if "NOT IN ('q1'" in sql: - return _Result([]) # Logged q1 is the only writer; the queue now speaks for it. - settings = self.writer_settings - return _Result( - [ - ( - "node", - "writer", - "async_insert" in settings, - settings.get("async_insert", ""), - "wait_for_async_insert" in settings, - settings.get("wait_for_async_insert", ""), - 3600, - 1, - ) - ] - ) - if "query_log" in sql: - return _Result([("node", "q1", "writer", bool(self.writer_settings))]) - if "system.settings_profiles" in sql: - return _Result([("node", "writer_profile", 0, ["writer"], [])]) - if "system.settings_profile_elements" in sql: - return _Result( - [ - ( - "node", - "writer_profile", - None, - None, - "wait_for_async_insert", - self.profile_value, - None, - ) - ] - ) - if "system.users" in sql: - return _Result([("node", "writer", 0, [], [])]) - if "system.role_grants" in sql: - return _Result([]) - if "name, default FROM" in sql: - return _Result([]) - if "system.settings " in sql: - return _Result([("0",)]) - raise AssertionError(sql) - - -def _table(name="t", engine="MergeTree", partition="toYYYYMM(day)"): - return TableDefinition(name, engine, [ColumnDefinition("id", "UInt64")], ["id"], partition) - - -def _codes(result): - return {finding.code: finding for finding in result.findings} - - -def test_inherited_profile_refuses_unacknowledged_async_writer(): - client = _Client() - result = inspect_rebuild(client, RebuildRequest("db", _table(), _table("new"))) - assert _codes(result)["unacknowledged_async_writer"].severity == "refusal" - assert result.insert_rows_per_second == 1.0 - assert result.partition_parts == {"202601": 9} - assert _codes(result)["many_parts"].severity == "warning" - - -def test_unacknowledged_loss_requires_explicit_acknowledgement(): - result = inspect_rebuild( - _Client(), - RebuildRequest("db", _table(), _table("new"), allow_unacknowledged_async_loss=True), - ) - assert _codes(result)["unacknowledged_async_writer"].severity == "warning" - - -def test_unknown_profile_setting_fails_closed(): - result = inspect_rebuild( - _Client(profile_value=None), RebuildRequest("db", _table(), _table("new")) - ) - assert _codes(result)["writer_settings_unknown"].severity == "refusal" - - -def test_transfer_pair_mismatch_not_intentional_row_copy(): - source = _table() - replacement = _table("new", "ReplacingMergeTree") - changed = _table("stage", "ReplicatedMergeTree") - result = inspect_rebuild( - _Client(writer_settings={"async_insert": "0", "wait_for_async_insert": "1"}), - RebuildRequest("db", source, replacement, transfer_pairs=((replacement, changed),)), - ) - assert _codes(result)["transfer_structure_mismatch"].details["differences"] == ["engine"] - assert "source_target_engine_mismatch" not in _codes(result) - - -def test_partition_key_and_mutation_and_shards_refuse(): - request = RebuildRequest("db", _table(), _table("new", partition="day"), cluster="replicas") - result = inspect_rebuild(_Client(mutation=True, shards=2), request) - codes = _codes(result) - assert {"partition_key_change", "unfinished_mutations", "sharded_cluster"} <= codes.keys() - assert all( - codes[code].severity == "refusal" - for code in ("partition_key_change", "unfinished_mutations", "sharded_cluster") - ) - - -def test_cloud_rebuild_requires_declared_single_service_writers(): - writer = {"async_insert": "0", "wait_for_async_insert": "1"} - shared = _table(engine="SharedReplacingMergeTree") - undeclared = inspect_rebuild( - _Client(writer_settings=writer), RebuildRequest("db", shared, _table("new")) - ) - assert _codes(undeclared)["cloud_writers_undeclared"].severity == "refusal" - declared = inspect_rebuild( - _Client(writer_settings=writer), - RebuildRequest("db", shared, _table("new"), writers_on_this_service_only=True), - ) - assert "cloud_writers_undeclared" not in _codes(declared) - assert _codes(declared)["cloud_writers_declared"].severity == "warning" - self_hosted = inspect_rebuild( - _Client(writer_settings=writer), - RebuildRequest("db", _table(engine="ReplicatedReplacingMergeTree"), _table("new")), - ) - assert not {"cloud_writers_undeclared", "cloud_writers_declared"} & _codes(self_hosted).keys() - - -def test_host_without_part_log_refuses_because_stranded_rows_would_go_unseen(): - class NoPartLog(_Client): - def query(self, sql): - if "name = 'part_log'" in sql: - return _Result([("node", 1), ("other", 0)]) - return super().query(sql) - - result = inspect_rebuild( - NoPartLog(writer_settings={"async_insert": "0", "wait_for_async_insert": "1"}), - RebuildRequest("db", _table(), _table("new")), - ) - refusal = _codes(result)["part_log_unavailable"] - assert refusal.severity == "refusal" and refusal.details == {"hosts": ["other"]} - - -def test_query_failure_is_not_silently_converted_to_empty_evidence(): - class Broken(_Client): - def query(self, sql): - if "query_log" in sql: - raise PermissionError("No query_log privilege") - return super().query(sql) - - with pytest.raises(PermissionError, match="query_log"): - inspect_rebuild(Broken(), RebuildRequest("db", _table(), _table("new"))) - - -def test_newly_queued_writer_without_log_evidence_requires_opt_in(): - class Queued(_Client): - def query(self, sql): - if "system.asynchronous_inserts" in sql: - return _Result([("node", ["unlogged-async-query"])]) - return super().query(sql) - - client = Queued(writer_settings={"async_insert": "0", "wait_for_async_insert": "1"}) - refused = inspect_rebuild(client, RebuildRequest("db", _table(), _table("new"))) - assert _codes(refused)["writer_settings_unknown"].severity == "refusal" - allowed = inspect_rebuild( - client, RebuildRequest("db", _table(), _table("new"), allow_unacknowledged_async_loss=True) - ) - assert _codes(allowed)["writer_settings_unknown"].severity == "warning" - - -@pytest.mark.parametrize("wait", ["0", "1"]) -@pytest.mark.parametrize("logged", [False, True]) -def test_active_queued_writer_uses_per_query_settings_before_log_flush(wait, logged): - query_id = "q1" if logged else "waiting" - - class Waiting(_Client): - def query(self, sql): - if "system.asynchronous_inserts" in sql: - return _Result([("node", [query_id])]) - if "system.processes" in sql: - return _Result( - [ - ( - "node", - query_id, - "writer", - {"async_insert": "1", "wait_for_async_insert": wait}, - ) - ] - ) - return super().query(sql) - - result = inspect_rebuild( - Waiting( - writer_settings={} if logged else {"async_insert": "0", "wait_for_async_insert": "1"} - ), - RebuildRequest("db", _table(), _table("new")), - ) - assert "writer_settings_unknown" not in _codes(result) - assert any(finding.severity == "refusal" for finding in result.findings) == (wait == "0") - if wait == "0": - assert _codes(result)["unacknowledged_async_writer"].severity == "refusal" - - -def test_another_hosts_process_cannot_prove_queued_writer_settings(): - class WrongHost(_Client): - def query(self, sql): - if "system.asynchronous_inserts" in sql: - return _Result([("node", ["waiting"])]) - if "system.processes" in sql: - return _Result( - [ - ( - "other", - "waiting", - "writer", - {"async_insert": "1", "wait_for_async_insert": "1"}, - ) - ] - ) - return super().query(sql) - - result = inspect_rebuild( - WrongHost(writer_settings={"async_insert": "0", "wait_for_async_insert": "1"}), - RebuildRequest("db", _table(), _table("new")), - ) - assert _codes(result)["writer_settings_unknown"].severity == "refusal" - - -def test_plain_helpers_transfer_within_a_replication_family_only(): - source = _table(engine="ReplicatedReplacingMergeTree('/source', 'r1', id)") - helper = _table("snap", "ReplicatedMergeTree('/snap', 'r2')") - client = _Client(writer_settings={"async_insert": "0", "wait_for_async_insert": "1"}) - result = inspect_rebuild( - client, RebuildRequest("db", source, _table("new"), transfer_pairs=((source, helper),)) - ) - assert "transfer_structure_mismatch" not in _codes(result) - helper.engine = "MergeTree" - mismatch = inspect_rebuild( - client, RebuildRequest("db", source, _table("new"), transfer_pairs=((source, helper),)) - ) - assert _codes(mismatch)["transfer_structure_mismatch"].details["differences"] == ["engine"] - - -def test_safe_profile_does_not_prove_unlogged_queued_insert_is_acknowledged(): - class HiddenOverride(_Client): - def query(self, sql): - if "system.asynchronous_inserts" in sql: - return _Result([("node", ["q1"])]) - if "system.settings_profile_elements" in sql: - return _Result( - [ - ("node", "writer_profile", None, None, "async_insert", "0", None), - ("node", "writer_profile", None, None, "wait_for_async_insert", "1", None), - ] - ) - return super().query(sql) - - result = inspect_rebuild( - HiddenOverride(writer_settings={}), RebuildRequest("db", _table(), _table("new")) - ) - assert _codes(result)["writer_settings_unknown"].severity == "refusal" diff --git a/tests/test_reruns.py b/tests/test_reruns.py new file mode 100644 index 0000000..e599aed --- /dev/null +++ b/tests/test_reruns.py @@ -0,0 +1,123 @@ +"""Rerunning an interrupted step: which steps ch-migrate vouches for, and the marker table.""" + +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from ch_migrate.waiting.reruns import InterruptedStep, RerunGuard, Step, unvouched_statements + + +def test_a_first_run_records_the_step_before_it_runs(tmp_path): + client = _Client(started=0) + _guard(client).begin(_step(tmp_path, "op.execute('SELECT 1')")) + assert any(sql.startswith("INSERT INTO `db`.`_ch_migrate_in_progress`") for sql in client.sent) + + +def test_an_interrupted_step_the_gate_passes_runs_again(tmp_path, caplog): + client = _Client(started=1) + step = _step(tmp_path, "op.execute('ALTER TABLE t ADD COLUMN IF NOT EXISTS c UInt8')") + _guard(client).begin(step) + assert "was interrupted on an earlier run; running it again" in caplog.text + assert any(sql.startswith("INSERT INTO") for sql in client.sent) + + +def test_an_interrupted_step_with_a_waiver_stops(tmp_path): + client = _Client(started=1) + step = _step( + tmp_path, + "# ch-migrate: allow-non-idempotent counted once per deploy\n" + " op.execute('ALTER TABLE t UPDATE n = n + 1 WHERE 1')", + ) + with pytest.raises(InterruptedStep) as stopped: + _guard(client).begin(step) + assert [item.why for item in stopped.value.unvouched] == ["waiver: counted once per deploy"] + assert "--rerun-interrupted bbbbbbbb" in str(stopped.value) + assert not any(sql.startswith("INSERT INTO") for sql in client.sent) + + +def test_naming_the_revision_lets_an_interrupted_step_run_again(tmp_path): + client = _Client(started=1) + step = _step(tmp_path, "op.execute(statement)") + _guard(client, acknowledged={"bbbbbbbb"}).begin(step) + assert any(sql.startswith("INSERT INTO") for sql in client.sent) + + +def test_a_recorded_step_clears_every_marker_for_its_revision(tmp_path): + client = _Client(started=0) + _guard(client).finish(_step(tmp_path, "op.execute('SELECT 1')")) + assert client.sent == [ + "ALTER TABLE `db`.`_ch_migrate_in_progress` DELETE WHERE revision = " + "'bbbbbbbbbbbb' SETTINGS mutations_sync = 2" + ] + + +def test_downgrades_are_judged_by_the_same_rules(tmp_path): + path = _revision( + tmp_path, + "op.execute('CREATE TABLE IF NOT EXISTS t (x UInt8) ENGINE = Memory')", + downgrade="op.execute('INSERT INTO t SELECT 1')", + ) + assert unvouched_statements(path, "upgrade") == [] + assert [item.why for item in unvouched_statements(path, "downgrade")] == [ + "not safe to run twice" + ] + + +@pytest.mark.parametrize( + ("engine", "cluster", "expected"), + [ + ("Atomic", None, "ENGINE = MergeTree ORDER BY"), + ("Shared", None, "ENGINE = MergeTree ORDER BY"), + ("Replicated", None, "ENGINE = ReplicatedMergeTree ORDER BY"), + ( + "Atomic", + "main", + "ON CLUSTER `main` (revision String, direction LowCardinality(String), started " + "DateTime64(3) DEFAULT now64(3)) ENGINE = ReplicatedMergeTree(" + "'/clickhouse/ch_migrate/db/{shard}/_ch_migrate_in_progress', '{replica}')", + ), + ], +) +def test_the_marker_table_follows_the_deployment(engine, cluster, expected): + state = _state(engine, cluster) + assert expected in RerunGuard(_Client(started=0), state, frozenset())._create_sql() + + +class _Client: + def __init__(self, started: int): + self.started, self.sent = started, [] + + def command(self, sql, parameters=None, settings=None): + if sql.startswith("EXISTS TABLE"): + return 1 + if sql.startswith("SELECT count()"): + return self.started + self.sent.append(sql) + return None + + +def _guard(client, acknowledged=()): + return RerunGuard(client, _state("Atomic", None), frozenset(acknowledged)) + + +def _state(engine, cluster): + on_cluster = cluster if engine == "Atomic" else None + return SimpleNamespace( + database="db", database_engine=engine, cluster=cluster, on_cluster=on_cluster + ) + + +def _step(tmp_path: Path, upgrade: str) -> Step: + return Step("bbbbbbbbbbbb", "upgrade", _revision(tmp_path, upgrade)) + + +def _revision(tmp_path: Path, upgrade: str, downgrade: str = "pass") -> Path: + versions = tmp_path / "migrations" / "versions" + versions.mkdir(parents=True, exist_ok=True) + path = versions / "bbbb.py" + path.write_text( + "revision = 'bbbbbbbbbbbb'\ndown_revision = None\n\n" + f"def upgrade():\n {upgrade}\n\ndef downgrade():\n {downgrade}\n" + ) + return path diff --git a/tests/test_snapshot.py b/tests/test_snapshot.py index eee1d44..8665254 100644 --- a/tests/test_snapshot.py +++ b/tests/test_snapshot.py @@ -58,13 +58,15 @@ def runner(tmp_path: Path, monkeypatch): """Set up a CLI runner in a tmp project directory.""" monkeypatch.chdir(tmp_path) (tmp_path / "migrations" / "sql" / "snapshots").mkdir(parents=True) - (tmp_path / "config.yaml").write_text(""" + (tmp_path / "config.yaml").write_text( + """ environments: dev: host: localhost database: testdb user: test -""") +""" + ) monkeypatch.setenv("CH_DEV_PASSWORD", "pass") return CliRunner() @@ -179,6 +181,7 @@ def test_timestamped_dirs_no_overwrite(self, mock_schema, mock_client, runner, t # Ensure a second snapshot creates a new directory (different timestamp) import time + time.sleep(1.1) result2 = runner.invoke(main, ["snapshot", "dev"]) diff --git a/tests/test_statements.py b/tests/test_statements.py index fe403bc..fffabdf 100644 --- a/tests/test_statements.py +++ b/tests/test_statements.py @@ -127,6 +127,59 @@ def test_missing_referenced_file_is_not_silently_ignored(project): migration_statements(path) +def test_every_way_of_sending_sql_is_collected(project): + path = _revision( + project, + "def upgrade():\n" + " op.execute(text('UPDATE {db}.t SET x = 1 WHERE 1'))\n" + " op.get_bind().execute(sa.text(f'DELETE FROM {db}.t WHERE id = 1'))\n" + " op.get_bind().exec_driver_sql('ALTER TABLE {db}.t UPDATE x = 2 WHERE 1')\n", + ) + statements = migration_statements(path) + assert [(s.sql, s.line, s.readable) for s in statements] == [ + ("UPDATE {db}.t SET x = 1 WHERE 1", 4, True), + ("DELETE FROM {db}.t WHERE id = 1", 5, True), + ("ALTER TABLE {db}.t UPDATE x = 2 WHERE 1", 6, True), + ] + + +def test_sql_known_only_at_run_time_is_unreadable_not_skipped(project): + path = _revision( + project, + "def upgrade():\n" + " # ch-migrate: allow-non-idempotent built from a reviewed list\n" + " op.execute(statement)\n" + " run_sql(name)\n" + " connection.exec_driver_sql(build())\n", + ) + statements = migration_statements(path) + assert [(s.sql, s.line, s.readable) for s in statements] == [ + ("op.execute(statement)", 5, False), + ("run_sql(name)", 6, False), + ("connection.exec_driver_sql(build())", 7, False), + ] + assert statements[0].comments == ( + "ch-migrate: allow-non-idempotent built from a reviewed list", + ) + + +def test_unreadable_sql_needs_a_waiver_and_skips_sql_rules(project): + _revision( + project, + "def upgrade():\n" + " op.execute(statement)\n" + " # ch-migrate: allow-non-idempotent each table is filled once\n" + " op.execute(other)\n", + ) + report = lint_migrations(project / "migrations/versions") + assert [(r.rule, r.severity.value, r.line) for r in report.results] == [ + ("idempotency", "error", 4), + ("idempotency", "info", 6), + ] + assert "only exists at run time" in report.results[0].message + assert report.results[1].message == "Idempotency waiver: each table is filled once" + + def _revision(project: Path, body: str) -> Path: path = project / "migrations/versions/a.py" path.write_text("revision = 'a'\ndown_revision = None\n" + body) diff --git a/tests/test_status_command.py b/tests/test_status_command.py index c53e80e..afe3bda 100644 --- a/tests/test_status_command.py +++ b/tests/test_status_command.py @@ -1,7 +1,7 @@ """`ch-migrate status` is a report: CI runs it as a non-blocking check. -Sazabi's required "ClickHouse Migration Status" job relies on status exiting 0 when -the database is unreachable or migrations are pending; only a broken project fails. +A required CI status job can rely on status exiting 0 when the database is +unreachable or migrations are pending; only a broken project fails. """ import pytest diff --git a/tests/test_waiting.py b/tests/test_waiting.py index 1a51af4..6cab874 100644 --- a/tests/test_waiting.py +++ b/tests/test_waiting.py @@ -1,93 +1,77 @@ -"""Ownership SQL and deadline/completion boundaries that could silently replay work.""" +"""Waiting decisions without a server: the deadline, what counts as a write, and completion.""" + +from types import SimpleNamespace import pytest -from click.testing import CliRunner - -from ch_migrate.cli import main -from ch_migrate.waiting_mutations import MutationWaiter -from ch_migrate.waiting_sql import ( - is_session_or_read, - mutation_sql, - query_setting, - statement_table, -) -from ch_migrate.waiting_types import WaitBudget, WaitTimeout +from ch_migrate.waiting.budget import WaitBudget +from ch_migrate.waiting.display import WaitDisplay +from ch_migrate.waiting.errors import MutationFailed, WaitTimeout +from ch_migrate.waiting.mutations import MutationWaiter +from ch_migrate.waiting.sql import is_write -@pytest.mark.parametrize("timeout", ["0", "-1", "nan", "inf", "-inf"]) -def test_wait_timeout_rejects_invalid_values_before_project_access(tmp_path, monkeypatch, timeout): - # Click's public command signature, not a database mock, decides usage errors. - monkeypatch.chdir(tmp_path) - result = CliRunner().invoke(main, ["up", "not-configured", "--timeout", timeout]) - assert result.exit_code == 2 - assert "--timeout" in result.output +def test_deadline_is_shared_across_waits(monkeypatch): + now = [10.0] + monkeypatch.setattr("ch_migrate.waiting.budget.time.monotonic", lambda: now[0]) + budget = WaitBudget(timeout=2, started=now[0]) + now[0] = 11.5 + assert budget.elapsed == 1.5 + now[0] = 12.0 + with pytest.raises(WaitTimeout, match="second mutation"): + budget.check("second mutation") -def test_mutation_settings_override_ignores_comment_and_literal_keywords(): - sql = ( - "ALTER TABLE db.t UPDATE value = 'SETTINGS -- not a comment', settings=settings+1 " - "WHERE id=1 -- trailing predicate comment\n" - "SETTINGS /* policy */ mutations_sync=2, max_threads=3 -- trailing setting comment\n;" - ) - rewritten = mutation_sql(sql, "owned-token") - assert query_setting(rewritten, "mutations_sync") == "0" - assert query_setting(rewritten, "max_threads") == "3" - assert "'SETTINGS -- not a comment'" in rewritten - assert "settings=settings+1 WHERE id=1" in rewritten - assert "DELETE WHERE 0" in rewritten +@pytest.mark.parametrize( + ("sql", "write"), + [ + ("SELECT 1", False), + (" -- note\n select count() FROM t", False), + ("/* why */ SET max_threads = 1", False), + ("SHOW TABLES", False), + ("EXISTS TABLE t", False), + ("ALTER TABLE t UPDATE x = 1 WHERE 1", True), + ("INSERT INTO t SELECT 1", True), + ("WITH source AS (SELECT 1) INSERT INTO t SELECT * FROM source", True), + ("CREATE TABLE t (x UInt8) ENGINE = Memory", True), + ("", True), + ], +) +def test_only_reads_and_session_statements_skip_the_wait(sql, write): + assert is_write(sql) is write -def test_settings_column_is_not_a_settings_clause(): - rewritten = mutation_sql("ALTER TABLE db.t UPDATE settings=2 WHERE settings=1", "owned-token") - assert "UPDATE settings=2 WHERE settings=1" in rewritten - assert query_setting(rewritten, "mutations_sync") == "0" +def test_zero_parts_left_is_not_completion(): + waiter = _waiter([_row("h1", "t", "mutation_1.txt", done=0, parts=0)]) + with pytest.raises(WaitTimeout, match="mutation_1.txt host h1; parts_to_do=0"): + waiter.wait() -def test_qualified_identifiers_preserve_dots_inside_quoted_names(): - assert statement_table( - 'ALTER TABLE "data.base".`some.table` UPDATE x=1 WHERE 1', "default" - ) == ("data.base", "some.table") +def test_a_replica_without_the_mutation_is_still_pending(): + waiter = _waiter([_row("r1", "t", "0000000001", done=1)], hosts={"t": ("r1", "r2")}) + with pytest.raises(WaitTimeout, match="0000000001 host r2; not received yet"): + waiter.wait() -@pytest.mark.parametrize( - "sql,readonly", - [ - ("WITH 1 AS x SELECT x", True), - ("WITH source AS (SELECT 1) INSERT INTO t SELECT * FROM source", False), - ("INSERT INTO t SELECT 'SELECT'", False), - ("SET max_threads=3", True), - ], -) -def test_statement_tracking_distinguishes_reads_from_writes(sql, readonly): - assert is_session_or_read(sql) is readonly +def test_a_failed_mutation_stops_with_the_scoped_kill(): + waiter = _waiter([_row("h1", "t", "mutation_2.txt", reason="Cannot parse 'n/a'")]) + with pytest.raises(MutationFailed) as raised: + waiter.wait(before_run=True) + failure = raised.value.failure + assert failure.foreign and failure.reason == "Cannot parse 'n/a'" + assert failure.kill.endswith("AND table = 't' AND mutation_id = 'mutation_2.txt'") -def test_deadline_is_shared_across_waits(monkeypatch): - now = [10.0] - monkeypatch.setattr("ch_migrate.waiting_types.time.monotonic", lambda: now[0]) - budget = WaitBudget(timeout=2, started=now[0]) - now[0] = 11.5 - assert budget.elapsed == 1.5 - now[0] = 12.0 - with pytest.raises(WaitTimeout, match="second mutation"): - budget.check("second mutation") + +def _waiter(rows, hosts=None): + state = SimpleNamespace(database="db", health_cluster=None) + budget = WaitBudget(timeout=0, display=WaitDisplay()) + waiter = MutationWaiter(None, state, budget) + # The real wait loop runs against these stubs: the rows the server returns, and which + # hosts each table's mutations must finish on. + waiter._rows = lambda expected: [waiter._row(row) for row in rows] + waiter._resolve_hosts = lambda table: (hosts or {}).get(table) + return waiter -def test_zero_parts_is_not_mutation_completion(monkeypatch): - waiter = MutationWaiter(None, None, WaitBudget(timeout=0)) - receipt = { - "database": "db", - "table": "t", - "engine": "MergeTree", - "initiator": "host", - "token": "owned", - "uuids": {"host": "uuid"}, - "ids": {"host": ["mutation_1.txt"]}, - "done_hosts": [], - "baseline": {"host": []}, - } - monkeypatch.setattr(waiter, "_identities", lambda _: {"host": "uuid"}) - monkeypatch.setattr(waiter, "_rows", lambda _: [("host", "mutation_1.txt", "owned", 0, 0, "")]) - with pytest.raises(WaitTimeout, match="mutation_1.txt"): - waiter.wait(receipt, lambda: None) - assert receipt["done_hosts"] == [] +def _row(host, table, mutation, done=0, parts=1, reason=""): + return (host, table, mutation, done, parts, reason, []) diff --git a/tests/test_waiting_display.py b/tests/test_waiting_display.py new file mode 100644 index 0000000..0275674 --- /dev/null +++ b/tests/test_waiting_display.py @@ -0,0 +1,174 @@ +"""What `up` shows while it waits: the terminal bar, piped lines and failure reports.""" + +import io + +import pytest +from rich.console import Console + +from ch_migrate.migration_runner import Upgrade, _report_failure, _report_interrupt +from ch_migrate.runner import _Progress +from ch_migrate.sql import SqlStatementError +from ch_migrate.statements import MigrationStatement +from ch_migrate.waiting.display import HostParts, MutationProgress, WaitDisplay +from ch_migrate.waiting.errors import FailedMutation, MutationFailed, WaitTimeout +from ch_migrate.waiting.reruns import InterruptedStep, Step, Unvouched + +GIB = 1 << 30 + + +@pytest.fixture +def clock(monkeypatch): + now = [100.0] + monkeypatch.setattr("ch_migrate.waiting.display.time.monotonic", lambda: now[0]) + return now + + +def test_terminal_bar_weights_by_bytes_and_estimates_time(clock): + display, buffer = _display(terminal=True) + display.start_migration("3f9c2a1b backfill_status") + display.update(_progress(_mutation(41))) + clock[0] = 172.0 + display.update(_progress(_mutation(18))) + assert "23.0 / 41.0 GiB · 23/41 parts · 1:12 · ~0:56 left" in buffer.getvalue() + assert "━" in buffer.getvalue() + clock[0] = 223.0 + display.complete("demo_dev.events") + assert ( + buffer.getvalue() + .rstrip() + .endswith("✓ 3f9c2a1b backfill_status mutation 7 · 41 parts · 41.0 GiB · 2:03") + ) + + +def test_piped_lines_start_then_every_tenth_or_thirty_seconds(clock): + display, buffer = _display(terminal=False) + display.start_migration("3f9c2a1b backfill_status") + for at, left in ((100, 40), (101, 38), (102, 36), (140, 35)): + clock[0] = float(at) + display.update(_progress(_mutation(left, sizes=False))) + clock[0] = 141.0 + display.complete("demo_dev.events") + assert buffer.getvalue().splitlines() == [ + " waiting on mutation 7 (demo_dev.events): 0/40 parts", + " waiting on mutation 7 (demo_dev.events): 4/40 parts · 0:02", + " waiting on mutation 7 (demo_dev.events): 5/40 parts · 0:40", + "✓ 3f9c2a1b backfill_status mutation 7 · 40 parts · 0:41", + ] + assert "\x1b" not in buffer.getvalue() and "\r" not in buffer.getvalue() + + +def test_unreadable_part_sizes_fall_back_to_parts_once(): + display, buffer = _display(terminal=True) + assert display.wants_bytes + display.parts_only("can't read system.parts (ACCESS_DENIED)") + display.parts_only("can't read system.parts (ACCESS_DENIED)") + assert buffer.getvalue().count("! Progress counts parts only") == 1 + assert not display.wants_bytes + + +def test_parts_only_bar_has_no_time_estimate(clock): + display, buffer = _display(terminal=True) + display.update(_progress(_mutation(10, sizes=False))) + clock[0] = 110.0 + display.update(_progress(_mutation(4, sizes=False))) + assert "6/10 parts" in buffer.getvalue() and "left" not in buffer.getvalue() + + +def test_each_replica_gets_its_own_row(clock): + display, buffer = _display(terminal=True) + display.update(_progress(_mutation(5, "ch-1"), _mutation(5, "ch-2"))) + clock[0] = 105.0 + display.update(_progress(_mutation(2, "ch-2"))) + text = buffer.getvalue() + assert "ch-1" in text and "ch-2" in text + display.complete("demo_dev.events") + assert "· 2 replicas" in buffer.getvalue().splitlines()[-1] + + +def test_failed_mutation_names_reason_retry_and_kill(capsys): + failure = FailedMutation( + "demo_dev.events", + "mutation_7.txt", + "h1", + "Cannot parse string 'n/a' as UInt32", + "KILL MUTATION WHERE database = 'demo_dev' AND table = 'events' AND mutation_id = 'mutation_7.txt'", + ) + error = _statement_error(MutationFailed("failed", failure)) + _report_failure(Upgrade("dev"), _current("3f9c2a1b widen_amount"), error) + err = capsys.readouterr().err + assert err.startswith( + "✗ 3f9c2a1b widen_amount (migrations/sql/x.up.sql:2) failed: mutation mutation_7.txt " + "on demo_dev.events failed on h1" + ) + assert "Cannot parse string 'n/a' as UInt32" in err + assert "keeps retrying a failed mutation" in err and failure.kill in err + assert ( + "If the cause is temporary" in err + and "then fixes the SQL and runs ch-migrate up dev" in err + ) + + +def test_timeout_says_the_work_continues_and_how_to_resume(capsys): + timeout = WaitTimeout( + "Timed out", "demo_dev.events mutation mutation_7.txt; parts_to_do=3", 3.0 + ) + _report_failure(Upgrade("dev"), _current("3f9c2a1b backfill"), _statement_error(timeout)) + err = capsys.readouterr().err + assert err.startswith("✗ Timed out after 3.0s waiting on 3f9c2a1b backfill") + assert "Pending: demo_dev.events mutation mutation_7.txt; parts_to_do=3" in err + assert "ch-migrate up dev again: it waits for that work to finish" in err + + +def test_interrupted_step_names_what_it_cannot_vouch_for(capsys): + statement = MigrationStatement( + "ALTER TABLE {db}.events UPDATE n = n + 1 WHERE 1", + "migrations/sql/bump.up.sql", + 2, + (), + "upgrade", + ) + stop = InterruptedStep( + Step("3f9c2a1b5d6e", "upgrade", None), [Unvouched(statement, "waiver: counted once")] + ) + _report_failure(Upgrade("dev"), _current("3f9c2a1b bump"), stop) + err = capsys.readouterr().err + assert err.startswith("✗ 3f9c2a1b was interrupted on an earlier run") + assert "migrations/sql/bump.up.sql:2 waiver: counted once: ALTER TABLE {db}.events" in err + assert "ch-migrate up dev --rerun-interrupted 3f9c2a1b" in err + + +def test_interrupt_says_the_work_continues(capsys): + _report_interrupt("dev", _current("3f9c2a1b backfill")) + err = capsys.readouterr().err + assert err.startswith("✗ Interrupted during 3f9c2a1b backfill") + assert "Run ch-migrate up dev again: it waits for that work to finish" in err + + +def _display(terminal): + buffer = io.StringIO() + console = Console(file=buffer, force_terminal=terminal, width=160, color_system=None) + return WaitDisplay(console), buffer + + +def _mutation(left, host="h1", sizes=True): + names = frozenset(f"all_{i}_{i}_0" for i in range(left)) + return HostParts(host, names, left, {name: GIB for name in names} if sizes else None) + + +def _progress(*hosts): + return MutationProgress("demo_dev.events", "7", hosts) + + +def _current(label): + progress = _Progress() + progress.current, progress.count = label, 1 + return progress + + +def _statement_error(cause): + try: + raise SqlStatementError( + f"migrations/sql/x.up.sql (statement 1 of 1, line 2): {cause}" + ) from cause + except SqlStatementError as error: + return error diff --git a/uv.lock b/uv.lock index 9bb6381..c88a6a1 100644 --- a/uv.lock +++ b/uv.lock @@ -3,8 +3,7 @@ revision = 3 requires-python = ">=3.10" resolution-markers = [ "python_full_version >= '3.14'", - "python_full_version >= '3.11' and python_full_version < '3.14'", - "python_full_version < '3.11'", + "python_full_version < '3.14'", ] [[package]] @@ -22,15 +21,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/3f/27/78a89b55b0904d222183164e079b4ca56208e94eff1d35ad1f1ad5be9b06/alembic-1.20.0-py3-none-any.whl", hash = "sha256:77eb101048d95f982c0353e9233404889dcd7a6fc244c107836c0e2fc9cf7d9d", size = 268719, upload-time = "2026-09-11T19:09:12.88Z" }, ] -[[package]] -name = "attrs" -version = "26.1.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/9a/8e/82a0fe20a541c03148528be8cac2408564a6c9a0cc7e9171802bc1d26985/attrs-26.1.0.tar.gz", hash = "sha256:d03ceb89cb322a8fd706d4fb91940737b6642aa36998fe130a9bc96c985eff32", size = 952055, upload-time = "2026-03-19T14:22:25.026Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/64/b4/17d4b0b2a2dc85a6df63d1157e028ed19f90d4cd97c36717afef2bc2f395/attrs-26.1.0-py3-none-any.whl", hash = "sha256:c647aa4a12dfbad9333ca4e71fe62ddc36f4e63b2d260a37a8b83d2f043ac309", size = 67548, upload-time = "2026-03-19T14:22:23.645Z" }, -] - [[package]] name = "backports-zstd" version = "1.7.0" @@ -214,7 +204,6 @@ dependencies = [ dev = [ { name = "black" }, { name = "isort" }, - { name = "jsonschema" }, { name = "mypy" }, { name = "pytest" }, { name = "pytest-cov" }, @@ -228,7 +217,6 @@ postgres = [ dev = [ { name = "black" }, { name = "isort" }, - { name = "jsonschema" }, { name = "mypy" }, { name = "pytest" }, { name = "pytest-cov" }, @@ -243,7 +231,6 @@ requires-dist = [ { name = "click", specifier = ">=8.1.0" }, { name = "clickhouse-connect", extras = ["alembic"], specifier = ">=1.9" }, { name = "isort", marker = "extra == 'dev'", specifier = ">=5.12.0" }, - { name = "jsonschema", marker = "extra == 'dev'", specifier = ">=4.18" }, { name = "mypy", marker = "extra == 'dev'", specifier = ">=1.5.0" }, { name = "psycopg2-binary", marker = "extra == 'postgres'", specifier = ">=2.9.0" }, { name = "pytest", marker = "extra == 'dev'", specifier = ">=7.4.0" }, @@ -260,7 +247,6 @@ provides-extras = ["dev", "postgres"] dev = [ { name = "black", specifier = ">=23.0.0" }, { name = "isort", specifier = ">=5.12.0" }, - { name = "jsonschema", specifier = ">=4.18" }, { name = "mypy", specifier = ">=1.5.0" }, { name = "pytest", specifier = ">=7.4.0" }, { name = "pytest-cov", specifier = ">=4.1.0" }, @@ -567,34 +553,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/31/b4/b9b800c45527aadd64d5b442f9b932b00648617eb5d63d2c7a6587b7cafc/jmespath-1.0.1-py3-none-any.whl", hash = "sha256:02e2e4cc71b5bcab88332eebf907519190dd9e6e82107fa7f83b1003a6252980", size = 20256, upload-time = "2022-06-17T18:00:10.251Z" }, ] -[[package]] -name = "jsonschema" -version = "4.26.0" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "attrs" }, - { name = "jsonschema-specifications" }, - { name = "referencing" }, - { name = "rpds-py", version = "0.30.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, - { name = "rpds-py", version = "2026.6.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/b3/fc/e067678238fa451312d4c62bf6e6cf5ec56375422aee02f9cb5f909b3047/jsonschema-4.26.0.tar.gz", hash = "sha256:0c26707e2efad8aa1bfc5b7ce170f3fccc2e4918ff85989ba9ffa9facb2be326", size = 366583, upload-time = "2026-01-07T13:41:07.246Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/69/90/f63fb5873511e014207a475e2bb4e8b2e570d655b00ac19a9a0ca0a385ee/jsonschema-4.26.0-py3-none-any.whl", hash = "sha256:d489f15263b8d200f8387e64b4c3a75f06629559fb73deb8fdfb525f2dab50ce", size = 90630, upload-time = "2026-01-07T13:41:05.306Z" }, -] - -[[package]] -name = "jsonschema-specifications" -version = "2025.9.1" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "referencing" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/19/74/a633ee74eb36c44aa6d1095e7cc5569bebf04342ee146178e2d36600708b/jsonschema_specifications-2025.9.1.tar.gz", hash = "sha256:b540987f239e745613c7a9176f3edb72b832a4ac465cf02712288397832b5e8d", size = 32855, upload-time = "2025-09-08T01:34:59.186Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/41/45/1a4ed80516f02155c51f51e8cedb3c1902296743db0bbc66608a0db2814f/jsonschema_specifications-2025.9.1-py3-none-any.whl", hash = "sha256:98802fee3a11ee76ecaca44429fda8a41bff98b00a0f2838151b113f210cc6fe", size = 18437, upload-time = "2025-09-08T01:34:57.871Z" }, -] - [[package]] name = "lz4" version = "4.4.5" @@ -1068,21 +1026,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/f1/12/de94a39c2ef588c7e6455cfbe7343d3b2dc9d6b6b2f40c4c6565744c873d/pyyaml-6.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:ebc55a14a21cb14062aa4162f906cd962b28e2e9ea38f9b4391244cd8de4ae0b", size = 149341, upload-time = "2025-09-25T21:32:56.828Z" }, ] -[[package]] -name = "referencing" -version = "0.37.0" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "attrs" }, - { name = "rpds-py", version = "0.30.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, - { name = "rpds-py", version = "2026.6.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" }, - { name = "typing-extensions", marker = "python_full_version < '3.13'" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/22/f5/df4e9027acead3ecc63e50fe1e36aca1523e1719559c499951bb4b53188f/referencing-0.37.0.tar.gz", hash = "sha256:44aefc3142c5b842538163acb373e24cce6632bd54bdb01b21ad5863489f50d8", size = 78036, upload-time = "2025-10-13T15:30:48.871Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/2c/58/ca301544e1fa93ed4f80d724bf5b194f6e4b945841c5bfd555878eea9fcb/referencing-0.37.0-py3-none-any.whl", hash = "sha256:381329a9f99628c9069361716891d34ad94af76e461dcb0335825aecc7692231", size = 26766, upload-time = "2025-10-13T15:30:47.625Z" }, -] - [[package]] name = "rich" version = "15.0.0" @@ -1096,258 +1039,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/82/3b/64d4899d73f91ba49a8c18a8ff3f0ea8f1c1d75481760df8c68ef5235bf5/rich-15.0.0-py3-none-any.whl", hash = "sha256:33bd4ef74232fb73fe9279a257718407f169c09b78a87ad3d296f548e27de0bb", size = 310654, upload-time = "2026-04-12T08:24:02.83Z" }, ] -[[package]] -name = "rpds-py" -version = "0.30.0" -source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version < '3.11'", -] -sdist = { url = "https://files.pythonhosted.org/packages/20/af/3f2f423103f1113b36230496629986e0ef7e199d2aa8392452b484b38ced/rpds_py-0.30.0.tar.gz", hash = "sha256:dd8ff7cf90014af0c0f787eea34794ebf6415242ee1d6fa91eaba725cc441e84", size = 69469, upload-time = "2025-11-30T20:24:38.837Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/06/0c/0c411a0ec64ccb6d104dcabe0e713e05e153a9a2c3c2bd2b32ce412166fe/rpds_py-0.30.0-cp310-cp310-macosx_10_12_x86_64.whl", hash = "sha256:679ae98e00c0e8d68a7fda324e16b90fd5260945b45d3b824c892cec9eea3288", size = 370490, upload-time = "2025-11-30T20:21:33.256Z" }, - { url = "https://files.pythonhosted.org/packages/19/6a/4ba3d0fb7297ebae71171822554abe48d7cab29c28b8f9f2c04b79988c05/rpds_py-0.30.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:4cc2206b76b4f576934f0ed374b10d7ca5f457858b157ca52064bdfc26b9fc00", size = 359751, upload-time = "2025-11-30T20:21:34.591Z" }, - { url = "https://files.pythonhosted.org/packages/cd/7c/e4933565ef7f7a0818985d87c15d9d273f1a649afa6a52ea35ad011195ea/rpds_py-0.30.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:389a2d49eded1896c3d48b0136ead37c48e221b391c052fba3f4055c367f60a6", size = 389696, upload-time = "2025-11-30T20:21:36.122Z" }, - { url = "https://files.pythonhosted.org/packages/5e/01/6271a2511ad0815f00f7ed4390cf2567bec1d4b1da39e2c27a41e6e3b4de/rpds_py-0.30.0-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:32c8528634e1bf7121f3de08fa85b138f4e0dc47657866630611b03967f041d7", size = 403136, upload-time = "2025-11-30T20:21:37.728Z" }, - { url = "https://files.pythonhosted.org/packages/55/64/c857eb7cd7541e9b4eee9d49c196e833128a55b89a9850a9c9ac33ccf897/rpds_py-0.30.0-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:f207f69853edd6f6700b86efb84999651baf3789e78a466431df1331608e5324", size = 524699, upload-time = "2025-11-30T20:21:38.92Z" }, - { url = "https://files.pythonhosted.org/packages/9c/ed/94816543404078af9ab26159c44f9e98e20fe47e2126d5d32c9d9948d10a/rpds_py-0.30.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:67b02ec25ba7a9e8fa74c63b6ca44cf5707f2fbfadae3ee8e7494297d56aa9df", size = 412022, upload-time = "2025-11-30T20:21:40.407Z" }, - { url = "https://files.pythonhosted.org/packages/61/b5/707f6cf0066a6412aacc11d17920ea2e19e5b2f04081c64526eb35b5c6e7/rpds_py-0.30.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0c0e95f6819a19965ff420f65578bacb0b00f251fefe2c8b23347c37174271f3", size = 390522, upload-time = "2025-11-30T20:21:42.17Z" }, - { url = "https://files.pythonhosted.org/packages/13/4e/57a85fda37a229ff4226f8cbcf09f2a455d1ed20e802ce5b2b4a7f5ed053/rpds_py-0.30.0-cp310-cp310-manylinux_2_31_riscv64.whl", hash = "sha256:a452763cc5198f2f98898eb98f7569649fe5da666c2dc6b5ddb10fde5a574221", size = 404579, upload-time = "2025-11-30T20:21:43.769Z" }, - { url = "https://files.pythonhosted.org/packages/f9/da/c9339293513ec680a721e0e16bf2bac3db6e5d7e922488de471308349bba/rpds_py-0.30.0-cp310-cp310-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:e0b65193a413ccc930671c55153a03ee57cecb49e6227204b04fae512eb657a7", size = 421305, upload-time = "2025-11-30T20:21:44.994Z" }, - { url = "https://files.pythonhosted.org/packages/f9/be/522cb84751114f4ad9d822ff5a1aa3c98006341895d5f084779b99596e5c/rpds_py-0.30.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:858738e9c32147f78b3ac24dc0edb6610000e56dc0f700fd5f651d0a0f0eb9ff", size = 572503, upload-time = "2025-11-30T20:21:46.91Z" }, - { url = "https://files.pythonhosted.org/packages/a2/9b/de879f7e7ceddc973ea6e4629e9b380213a6938a249e94b0cdbcc325bb66/rpds_py-0.30.0-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:da279aa314f00acbb803da1e76fa18666778e8a8f83484fba94526da5de2cba7", size = 598322, upload-time = "2025-11-30T20:21:48.709Z" }, - { url = "https://files.pythonhosted.org/packages/48/ac/f01fc22efec3f37d8a914fc1b2fb9bcafd56a299edbe96406f3053edea5a/rpds_py-0.30.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:7c64d38fb49b6cdeda16ab49e35fe0da2e1e9b34bc38bd78386530f218b37139", size = 560792, upload-time = "2025-11-30T20:21:50.024Z" }, - { url = "https://files.pythonhosted.org/packages/e2/da/4e2b19d0f131f35b6146425f846563d0ce036763e38913d917187307a671/rpds_py-0.30.0-cp310-cp310-win32.whl", hash = "sha256:6de2a32a1665b93233cde140ff8b3467bdb9e2af2b91079f0333a0974d12d464", size = 221901, upload-time = "2025-11-30T20:21:51.32Z" }, - { url = "https://files.pythonhosted.org/packages/96/cb/156d7a5cf4f78a7cc571465d8aec7a3c447c94f6749c5123f08438bcf7bc/rpds_py-0.30.0-cp310-cp310-win_amd64.whl", hash = "sha256:1726859cd0de969f88dc8673bdd954185b9104e05806be64bcd87badbe313169", size = 235823, upload-time = "2025-11-30T20:21:52.505Z" }, - { url = "https://files.pythonhosted.org/packages/4d/6e/f964e88b3d2abee2a82c1ac8366da848fce1c6d834dc2132c3fda3970290/rpds_py-0.30.0-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:a2bffea6a4ca9f01b3f8e548302470306689684e61602aa3d141e34da06cf425", size = 370157, upload-time = "2025-11-30T20:21:53.789Z" }, - { url = "https://files.pythonhosted.org/packages/94/ba/24e5ebb7c1c82e74c4e4f33b2112a5573ddc703915b13a073737b59b86e0/rpds_py-0.30.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:dc4f992dfe1e2bc3ebc7444f6c7051b4bc13cd8e33e43511e8ffd13bf407010d", size = 359676, upload-time = "2025-11-30T20:21:55.475Z" }, - { url = "https://files.pythonhosted.org/packages/84/86/04dbba1b087227747d64d80c3b74df946b986c57af0a9f0c98726d4d7a3b/rpds_py-0.30.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:422c3cb9856d80b09d30d2eb255d0754b23e090034e1deb4083f8004bd0761e4", size = 389938, upload-time = "2025-11-30T20:21:57.079Z" }, - { url = "https://files.pythonhosted.org/packages/42/bb/1463f0b1722b7f45431bdd468301991d1328b16cffe0b1c2918eba2c4eee/rpds_py-0.30.0-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:07ae8a593e1c3c6b82ca3292efbe73c30b61332fd612e05abee07c79359f292f", size = 402932, upload-time = "2025-11-30T20:21:58.47Z" }, - { url = "https://files.pythonhosted.org/packages/99/ee/2520700a5c1f2d76631f948b0736cdf9b0acb25abd0ca8e889b5c62ac2e3/rpds_py-0.30.0-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:12f90dd7557b6bd57f40abe7747e81e0c0b119bef015ea7726e69fe550e394a4", size = 525830, upload-time = "2025-11-30T20:21:59.699Z" }, - { url = "https://files.pythonhosted.org/packages/e0/ad/bd0331f740f5705cc555a5e17fdf334671262160270962e69a2bdef3bf76/rpds_py-0.30.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:99b47d6ad9a6da00bec6aabe5a6279ecd3c06a329d4aa4771034a21e335c3a97", size = 412033, upload-time = "2025-11-30T20:22:00.991Z" }, - { url = "https://files.pythonhosted.org/packages/f8/1e/372195d326549bb51f0ba0f2ecb9874579906b97e08880e7a65c3bef1a99/rpds_py-0.30.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:33f559f3104504506a44bb666b93a33f5d33133765b0c216a5bf2f1e1503af89", size = 390828, upload-time = "2025-11-30T20:22:02.723Z" }, - { url = "https://files.pythonhosted.org/packages/ab/2b/d88bb33294e3e0c76bc8f351a3721212713629ffca1700fa94979cb3eae8/rpds_py-0.30.0-cp311-cp311-manylinux_2_31_riscv64.whl", hash = "sha256:946fe926af6e44f3697abbc305ea168c2c31d3e3ef1058cf68f379bf0335a78d", size = 404683, upload-time = "2025-11-30T20:22:04.367Z" }, - { url = "https://files.pythonhosted.org/packages/50/32/c759a8d42bcb5289c1fac697cd92f6fe01a018dd937e62ae77e0e7f15702/rpds_py-0.30.0-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:495aeca4b93d465efde585977365187149e75383ad2684f81519f504f5c13038", size = 421583, upload-time = "2025-11-30T20:22:05.814Z" }, - { url = "https://files.pythonhosted.org/packages/2b/81/e729761dbd55ddf5d84ec4ff1f47857f4374b0f19bdabfcf929164da3e24/rpds_py-0.30.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:d9a0ca5da0386dee0655b4ccdf46119df60e0f10da268d04fe7cc87886872ba7", size = 572496, upload-time = "2025-11-30T20:22:07.713Z" }, - { url = "https://files.pythonhosted.org/packages/14/f6/69066a924c3557c9c30baa6ec3a0aa07526305684c6f86c696b08860726c/rpds_py-0.30.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:8d6d1cc13664ec13c1b84241204ff3b12f9bb82464b8ad6e7a5d3486975c2eed", size = 598669, upload-time = "2025-11-30T20:22:09.312Z" }, - { url = "https://files.pythonhosted.org/packages/5f/48/905896b1eb8a05630d20333d1d8ffd162394127b74ce0b0784ae04498d32/rpds_py-0.30.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:3896fa1be39912cf0757753826bc8bdc8ca331a28a7c4ae46b7a21280b06bb85", size = 561011, upload-time = "2025-11-30T20:22:11.309Z" }, - { url = "https://files.pythonhosted.org/packages/22/16/cd3027c7e279d22e5eb431dd3c0fbc677bed58797fe7581e148f3f68818b/rpds_py-0.30.0-cp311-cp311-win32.whl", hash = "sha256:55f66022632205940f1827effeff17c4fa7ae1953d2b74a8581baaefb7d16f8c", size = 221406, upload-time = "2025-11-30T20:22:13.101Z" }, - { url = "https://files.pythonhosted.org/packages/fa/5b/e7b7aa136f28462b344e652ee010d4de26ee9fd16f1bfd5811f5153ccf89/rpds_py-0.30.0-cp311-cp311-win_amd64.whl", hash = "sha256:a51033ff701fca756439d641c0ad09a41d9242fa69121c7d8769604a0a629825", size = 236024, upload-time = "2025-11-30T20:22:14.853Z" }, - { url = "https://files.pythonhosted.org/packages/14/a6/364bba985e4c13658edb156640608f2c9e1d3ea3c81b27aa9d889fff0e31/rpds_py-0.30.0-cp311-cp311-win_arm64.whl", hash = "sha256:47b0ef6231c58f506ef0b74d44e330405caa8428e770fec25329ed2cb971a229", size = 229069, upload-time = "2025-11-30T20:22:16.577Z" }, - { url = "https://files.pythonhosted.org/packages/03/e7/98a2f4ac921d82f33e03f3835f5bf3a4a40aa1bfdc57975e74a97b2b4bdd/rpds_py-0.30.0-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:a161f20d9a43006833cd7068375a94d035714d73a172b681d8881820600abfad", size = 375086, upload-time = "2025-11-30T20:22:17.93Z" }, - { url = "https://files.pythonhosted.org/packages/4d/a1/bca7fd3d452b272e13335db8d6b0b3ecde0f90ad6f16f3328c6fb150c889/rpds_py-0.30.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:6abc8880d9d036ecaafe709079969f56e876fcf107f7a8e9920ba6d5a3878d05", size = 359053, upload-time = "2025-11-30T20:22:19.297Z" }, - { url = "https://files.pythonhosted.org/packages/65/1c/ae157e83a6357eceff62ba7e52113e3ec4834a84cfe07fa4b0757a7d105f/rpds_py-0.30.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ca28829ae5f5d569bb62a79512c842a03a12576375d5ece7d2cadf8abe96ec28", size = 390763, upload-time = "2025-11-30T20:22:21.661Z" }, - { url = "https://files.pythonhosted.org/packages/d4/36/eb2eb8515e2ad24c0bd43c3ee9cd74c33f7ca6430755ccdb240fd3144c44/rpds_py-0.30.0-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:a1010ed9524c73b94d15919ca4d41d8780980e1765babf85f9a2f90d247153dd", size = 408951, upload-time = "2025-11-30T20:22:23.408Z" }, - { url = "https://files.pythonhosted.org/packages/d6/65/ad8dc1784a331fabbd740ef6f71ce2198c7ed0890dab595adb9ea2d775a1/rpds_py-0.30.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:f8d1736cfb49381ba528cd5baa46f82fdc65c06e843dab24dd70b63d09121b3f", size = 514622, upload-time = "2025-11-30T20:22:25.16Z" }, - { url = "https://files.pythonhosted.org/packages/63/8e/0cfa7ae158e15e143fe03993b5bcd743a59f541f5952e1546b1ac1b5fd45/rpds_py-0.30.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:d948b135c4693daff7bc2dcfc4ec57237a29bd37e60c2fabf5aff2bbacf3e2f1", size = 414492, upload-time = "2025-11-30T20:22:26.505Z" }, - { url = "https://files.pythonhosted.org/packages/60/1b/6f8f29f3f995c7ffdde46a626ddccd7c63aefc0efae881dc13b6e5d5bb16/rpds_py-0.30.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:47f236970bccb2233267d89173d3ad2703cd36a0e2a6e92d0560d333871a3d23", size = 394080, upload-time = "2025-11-30T20:22:27.934Z" }, - { url = "https://files.pythonhosted.org/packages/6d/d5/a266341051a7a3ca2f4b750a3aa4abc986378431fc2da508c5034d081b70/rpds_py-0.30.0-cp312-cp312-manylinux_2_31_riscv64.whl", hash = "sha256:2e6ecb5a5bcacf59c3f912155044479af1d0b6681280048b338b28e364aca1f6", size = 408680, upload-time = "2025-11-30T20:22:29.341Z" }, - { url = "https://files.pythonhosted.org/packages/10/3b/71b725851df9ab7a7a4e33cf36d241933da66040d195a84781f49c50490c/rpds_py-0.30.0-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:a8fa71a2e078c527c3e9dc9fc5a98c9db40bcc8a92b4e8858e36d329f8684b51", size = 423589, upload-time = "2025-11-30T20:22:31.469Z" }, - { url = "https://files.pythonhosted.org/packages/00/2b/e59e58c544dc9bd8bd8384ecdb8ea91f6727f0e37a7131baeff8d6f51661/rpds_py-0.30.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:73c67f2db7bc334e518d097c6d1e6fed021bbc9b7d678d6cc433478365d1d5f5", size = 573289, upload-time = "2025-11-30T20:22:32.997Z" }, - { url = "https://files.pythonhosted.org/packages/da/3e/a18e6f5b460893172a7d6a680e86d3b6bc87a54c1f0b03446a3c8c7b588f/rpds_py-0.30.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:5ba103fb455be00f3b1c2076c9d4264bfcb037c976167a6047ed82f23153f02e", size = 599737, upload-time = "2025-11-30T20:22:34.419Z" }, - { url = "https://files.pythonhosted.org/packages/5c/e2/714694e4b87b85a18e2c243614974413c60aa107fd815b8cbc42b873d1d7/rpds_py-0.30.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:7cee9c752c0364588353e627da8a7e808a66873672bcb5f52890c33fd965b394", size = 563120, upload-time = "2025-11-30T20:22:35.903Z" }, - { url = "https://files.pythonhosted.org/packages/6f/ab/d5d5e3bcedb0a77f4f613706b750e50a5a3ba1c15ccd3665ecc636c968fd/rpds_py-0.30.0-cp312-cp312-win32.whl", hash = "sha256:1ab5b83dbcf55acc8b08fc62b796ef672c457b17dbd7820a11d6c52c06839bdf", size = 223782, upload-time = "2025-11-30T20:22:37.271Z" }, - { url = "https://files.pythonhosted.org/packages/39/3b/f786af9957306fdc38a74cef405b7b93180f481fb48453a114bb6465744a/rpds_py-0.30.0-cp312-cp312-win_amd64.whl", hash = "sha256:a090322ca841abd453d43456ac34db46e8b05fd9b3b4ac0c78bcde8b089f959b", size = 240463, upload-time = "2025-11-30T20:22:39.021Z" }, - { url = "https://files.pythonhosted.org/packages/f3/d2/b91dc748126c1559042cfe41990deb92c4ee3e2b415f6b5234969ffaf0cc/rpds_py-0.30.0-cp312-cp312-win_arm64.whl", hash = "sha256:669b1805bd639dd2989b281be2cfd951c6121b65e729d9b843e9639ef1fd555e", size = 230868, upload-time = "2025-11-30T20:22:40.493Z" }, - { url = "https://files.pythonhosted.org/packages/ed/dc/d61221eb88ff410de3c49143407f6f3147acf2538c86f2ab7ce65ae7d5f9/rpds_py-0.30.0-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:f83424d738204d9770830d35290ff3273fbb02b41f919870479fab14b9d303b2", size = 374887, upload-time = "2025-11-30T20:22:41.812Z" }, - { url = "https://files.pythonhosted.org/packages/fd/32/55fb50ae104061dbc564ef15cc43c013dc4a9f4527a1f4d99baddf56fe5f/rpds_py-0.30.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:e7536cd91353c5273434b4e003cbda89034d67e7710eab8761fd918ec6c69cf8", size = 358904, upload-time = "2025-11-30T20:22:43.479Z" }, - { url = "https://files.pythonhosted.org/packages/58/70/faed8186300e3b9bdd138d0273109784eea2396c68458ed580f885dfe7ad/rpds_py-0.30.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2771c6c15973347f50fece41fc447c054b7ac2ae0502388ce3b6738cd366e3d4", size = 389945, upload-time = "2025-11-30T20:22:44.819Z" }, - { url = "https://files.pythonhosted.org/packages/bd/a8/073cac3ed2c6387df38f71296d002ab43496a96b92c823e76f46b8af0543/rpds_py-0.30.0-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:0a59119fc6e3f460315fe9d08149f8102aa322299deaa5cab5b40092345c2136", size = 407783, upload-time = "2025-11-30T20:22:46.103Z" }, - { url = "https://files.pythonhosted.org/packages/77/57/5999eb8c58671f1c11eba084115e77a8899d6e694d2a18f69f0ba471ec8b/rpds_py-0.30.0-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:76fec018282b4ead0364022e3c54b60bf368b9d926877957a8624b58419169b7", size = 515021, upload-time = "2025-11-30T20:22:47.458Z" }, - { url = "https://files.pythonhosted.org/packages/e0/af/5ab4833eadc36c0a8ed2bc5c0de0493c04f6c06de223170bd0798ff98ced/rpds_py-0.30.0-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:692bef75a5525db97318e8cd061542b5a79812d711ea03dbc1f6f8dbb0c5f0d2", size = 414589, upload-time = "2025-11-30T20:22:48.872Z" }, - { url = "https://files.pythonhosted.org/packages/b7/de/f7192e12b21b9e9a68a6d0f249b4af3fdcdff8418be0767a627564afa1f1/rpds_py-0.30.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9027da1ce107104c50c81383cae773ef5c24d296dd11c99e2629dbd7967a20c6", size = 394025, upload-time = "2025-11-30T20:22:50.196Z" }, - { url = "https://files.pythonhosted.org/packages/91/c4/fc70cd0249496493500e7cc2de87504f5aa6509de1e88623431fec76d4b6/rpds_py-0.30.0-cp313-cp313-manylinux_2_31_riscv64.whl", hash = "sha256:9cf69cdda1f5968a30a359aba2f7f9aa648a9ce4b580d6826437f2b291cfc86e", size = 408895, upload-time = "2025-11-30T20:22:51.87Z" }, - { url = "https://files.pythonhosted.org/packages/58/95/d9275b05ab96556fefff73a385813eb66032e4c99f411d0795372d9abcea/rpds_py-0.30.0-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:a4796a717bf12b9da9d3ad002519a86063dcac8988b030e405704ef7d74d2d9d", size = 422799, upload-time = "2025-11-30T20:22:53.341Z" }, - { url = "https://files.pythonhosted.org/packages/06/c1/3088fc04b6624eb12a57eb814f0d4997a44b0d208d6cace713033ff1a6ba/rpds_py-0.30.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:5d4c2aa7c50ad4728a094ebd5eb46c452e9cb7edbfdb18f9e1221f597a73e1e7", size = 572731, upload-time = "2025-11-30T20:22:54.778Z" }, - { url = "https://files.pythonhosted.org/packages/d8/42/c612a833183b39774e8ac8fecae81263a68b9583ee343db33ab571a7ce55/rpds_py-0.30.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:ba81a9203d07805435eb06f536d95a266c21e5b2dfbf6517748ca40c98d19e31", size = 599027, upload-time = "2025-11-30T20:22:56.212Z" }, - { url = "https://files.pythonhosted.org/packages/5f/60/525a50f45b01d70005403ae0e25f43c0384369ad24ffe46e8d9068b50086/rpds_py-0.30.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:945dccface01af02675628334f7cf49c2af4c1c904748efc5cf7bbdf0b579f95", size = 563020, upload-time = "2025-11-30T20:22:58.2Z" }, - { url = "https://files.pythonhosted.org/packages/0b/5d/47c4655e9bcd5ca907148535c10e7d489044243cc9941c16ed7cd53be91d/rpds_py-0.30.0-cp313-cp313-win32.whl", hash = "sha256:b40fb160a2db369a194cb27943582b38f79fc4887291417685f3ad693c5a1d5d", size = 223139, upload-time = "2025-11-30T20:23:00.209Z" }, - { url = "https://files.pythonhosted.org/packages/f2/e1/485132437d20aa4d3e1d8b3fb5a5e65aa8139f1e097080c2a8443201742c/rpds_py-0.30.0-cp313-cp313-win_amd64.whl", hash = "sha256:806f36b1b605e2d6a72716f321f20036b9489d29c51c91f4dd29a3e3afb73b15", size = 240224, upload-time = "2025-11-30T20:23:02.008Z" }, - { url = "https://files.pythonhosted.org/packages/24/95/ffd128ed1146a153d928617b0ef673960130be0009c77d8fbf0abe306713/rpds_py-0.30.0-cp313-cp313-win_arm64.whl", hash = "sha256:d96c2086587c7c30d44f31f42eae4eac89b60dabbac18c7669be3700f13c3ce1", size = 230645, upload-time = "2025-11-30T20:23:03.43Z" }, - { url = "https://files.pythonhosted.org/packages/ff/1b/b10de890a0def2a319a2626334a7f0ae388215eb60914dbac8a3bae54435/rpds_py-0.30.0-cp313-cp313t-macosx_10_12_x86_64.whl", hash = "sha256:eb0b93f2e5c2189ee831ee43f156ed34e2a89a78a66b98cadad955972548be5a", size = 364443, upload-time = "2025-11-30T20:23:04.878Z" }, - { url = "https://files.pythonhosted.org/packages/0d/bf/27e39f5971dc4f305a4fb9c672ca06f290f7c4e261c568f3dea16a410d47/rpds_py-0.30.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:922e10f31f303c7c920da8981051ff6d8c1a56207dbdf330d9047f6d30b70e5e", size = 353375, upload-time = "2025-11-30T20:23:06.342Z" }, - { url = "https://files.pythonhosted.org/packages/40/58/442ada3bba6e8e6615fc00483135c14a7538d2ffac30e2d933ccf6852232/rpds_py-0.30.0-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cdc62c8286ba9bf7f47befdcea13ea0e26bf294bda99758fd90535cbaf408000", size = 383850, upload-time = "2025-11-30T20:23:07.825Z" }, - { url = "https://files.pythonhosted.org/packages/14/14/f59b0127409a33c6ef6f5c1ebd5ad8e32d7861c9c7adfa9a624fc3889f6c/rpds_py-0.30.0-cp313-cp313t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:47f9a91efc418b54fb8190a6b4aa7813a23fb79c51f4bb84e418f5476c38b8db", size = 392812, upload-time = "2025-11-30T20:23:09.228Z" }, - { url = "https://files.pythonhosted.org/packages/b3/66/e0be3e162ac299b3a22527e8913767d869e6cc75c46bd844aa43fb81ab62/rpds_py-0.30.0-cp313-cp313t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:1f3587eb9b17f3789ad50824084fa6f81921bbf9a795826570bda82cb3ed91f2", size = 517841, upload-time = "2025-11-30T20:23:11.186Z" }, - { url = "https://files.pythonhosted.org/packages/3d/55/fa3b9cf31d0c963ecf1ba777f7cf4b2a2c976795ac430d24a1f43d25a6ba/rpds_py-0.30.0-cp313-cp313t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:39c02563fc592411c2c61d26b6c5fe1e51eaa44a75aa2c8735ca88b0d9599daa", size = 408149, upload-time = "2025-11-30T20:23:12.864Z" }, - { url = "https://files.pythonhosted.org/packages/60/ca/780cf3b1a32b18c0f05c441958d3758f02544f1d613abf9488cd78876378/rpds_py-0.30.0-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:51a1234d8febafdfd33a42d97da7a43f5dcb120c1060e352a3fbc0c6d36e2083", size = 383843, upload-time = "2025-11-30T20:23:14.638Z" }, - { url = "https://files.pythonhosted.org/packages/82/86/d5f2e04f2aa6247c613da0c1dd87fcd08fa17107e858193566048a1e2f0a/rpds_py-0.30.0-cp313-cp313t-manylinux_2_31_riscv64.whl", hash = "sha256:eb2c4071ab598733724c08221091e8d80e89064cd472819285a9ab0f24bcedb9", size = 396507, upload-time = "2025-11-30T20:23:16.105Z" }, - { url = "https://files.pythonhosted.org/packages/4b/9a/453255d2f769fe44e07ea9785c8347edaf867f7026872e76c1ad9f7bed92/rpds_py-0.30.0-cp313-cp313t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:6bdfdb946967d816e6adf9a3d8201bfad269c67efe6cefd7093ef959683c8de0", size = 414949, upload-time = "2025-11-30T20:23:17.539Z" }, - { url = "https://files.pythonhosted.org/packages/a3/31/622a86cdc0c45d6df0e9ccb6becdba5074735e7033c20e401a6d9d0e2ca0/rpds_py-0.30.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:c77afbd5f5250bf27bf516c7c4a016813eb2d3e116139aed0096940c5982da94", size = 565790, upload-time = "2025-11-30T20:23:19.029Z" }, - { url = "https://files.pythonhosted.org/packages/1c/5d/15bbf0fb4a3f58a3b1c67855ec1efcc4ceaef4e86644665fff03e1b66d8d/rpds_py-0.30.0-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:61046904275472a76c8c90c9ccee9013d70a6d0f73eecefd38c1ae7c39045a08", size = 590217, upload-time = "2025-11-30T20:23:20.885Z" }, - { url = "https://files.pythonhosted.org/packages/6d/61/21b8c41f68e60c8cc3b2e25644f0e3681926020f11d06ab0b78e3c6bbff1/rpds_py-0.30.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:4c5f36a861bc4b7da6516dbdf302c55313afa09b81931e8280361a4f6c9a2d27", size = 555806, upload-time = "2025-11-30T20:23:22.488Z" }, - { url = "https://files.pythonhosted.org/packages/f9/39/7e067bb06c31de48de3eb200f9fc7c58982a4d3db44b07e73963e10d3be9/rpds_py-0.30.0-cp313-cp313t-win32.whl", hash = "sha256:3d4a69de7a3e50ffc214ae16d79d8fbb0922972da0356dcf4d0fdca2878559c6", size = 211341, upload-time = "2025-11-30T20:23:24.449Z" }, - { url = "https://files.pythonhosted.org/packages/0a/4d/222ef0b46443cf4cf46764d9c630f3fe4abaa7245be9417e56e9f52b8f65/rpds_py-0.30.0-cp313-cp313t-win_amd64.whl", hash = "sha256:f14fc5df50a716f7ece6a80b6c78bb35ea2ca47c499e422aa4463455dd96d56d", size = 225768, upload-time = "2025-11-30T20:23:25.908Z" }, - { url = "https://files.pythonhosted.org/packages/86/81/dad16382ebbd3d0e0328776d8fd7ca94220e4fa0798d1dc5e7da48cb3201/rpds_py-0.30.0-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:68f19c879420aa08f61203801423f6cd5ac5f0ac4ac82a2368a9fcd6a9a075e0", size = 362099, upload-time = "2025-11-30T20:23:27.316Z" }, - { url = "https://files.pythonhosted.org/packages/2b/60/19f7884db5d5603edf3c6bce35408f45ad3e97e10007df0e17dd57af18f8/rpds_py-0.30.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:ec7c4490c672c1a0389d319b3a9cfcd098dcdc4783991553c332a15acf7249be", size = 353192, upload-time = "2025-11-30T20:23:29.151Z" }, - { url = "https://files.pythonhosted.org/packages/bf/c4/76eb0e1e72d1a9c4703c69607cec123c29028bff28ce41588792417098ac/rpds_py-0.30.0-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f251c812357a3fed308d684a5079ddfb9d933860fc6de89f2b7ab00da481e65f", size = 384080, upload-time = "2025-11-30T20:23:30.785Z" }, - { url = "https://files.pythonhosted.org/packages/72/87/87ea665e92f3298d1b26d78814721dc39ed8d2c74b86e83348d6b48a6f31/rpds_py-0.30.0-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:ac98b175585ecf4c0348fd7b29c3864bda53b805c773cbf7bfdaffc8070c976f", size = 394841, upload-time = "2025-11-30T20:23:32.209Z" }, - { url = "https://files.pythonhosted.org/packages/77/ad/7783a89ca0587c15dcbf139b4a8364a872a25f861bdb88ed99f9b0dec985/rpds_py-0.30.0-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:3e62880792319dbeb7eb866547f2e35973289e7d5696c6e295476448f5b63c87", size = 516670, upload-time = "2025-11-30T20:23:33.742Z" }, - { url = "https://files.pythonhosted.org/packages/5b/3c/2882bdac942bd2172f3da574eab16f309ae10a3925644e969536553cb4ee/rpds_py-0.30.0-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:4e7fc54e0900ab35d041b0601431b0a0eb495f0851a0639b6ef90f7741b39a18", size = 408005, upload-time = "2025-11-30T20:23:35.253Z" }, - { url = "https://files.pythonhosted.org/packages/ce/81/9a91c0111ce1758c92516a3e44776920b579d9a7c09b2b06b642d4de3f0f/rpds_py-0.30.0-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:47e77dc9822d3ad616c3d5759ea5631a75e5809d5a28707744ef79d7a1bcfcad", size = 382112, upload-time = "2025-11-30T20:23:36.842Z" }, - { url = "https://files.pythonhosted.org/packages/cf/8e/1da49d4a107027e5fbc64daeab96a0706361a2918da10cb41769244b805d/rpds_py-0.30.0-cp314-cp314-manylinux_2_31_riscv64.whl", hash = "sha256:b4dc1a6ff022ff85ecafef7979a2c6eb423430e05f1165d6688234e62ba99a07", size = 399049, upload-time = "2025-11-30T20:23:38.343Z" }, - { url = "https://files.pythonhosted.org/packages/df/5a/7ee239b1aa48a127570ec03becbb29c9d5a9eb092febbd1699d567cae859/rpds_py-0.30.0-cp314-cp314-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:4559c972db3a360808309e06a74628b95eaccbf961c335c8fe0d590cf587456f", size = 415661, upload-time = "2025-11-30T20:23:40.263Z" }, - { url = "https://files.pythonhosted.org/packages/70/ea/caa143cf6b772f823bc7929a45da1fa83569ee49b11d18d0ada7f5ee6fd6/rpds_py-0.30.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:0ed177ed9bded28f8deb6ab40c183cd1192aa0de40c12f38be4d59cd33cb5c65", size = 565606, upload-time = "2025-11-30T20:23:42.186Z" }, - { url = "https://files.pythonhosted.org/packages/64/91/ac20ba2d69303f961ad8cf55bf7dbdb4763f627291ba3d0d7d67333cced9/rpds_py-0.30.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:ad1fa8db769b76ea911cb4e10f049d80bf518c104f15b3edb2371cc65375c46f", size = 591126, upload-time = "2025-11-30T20:23:44.086Z" }, - { url = "https://files.pythonhosted.org/packages/21/20/7ff5f3c8b00c8a95f75985128c26ba44503fb35b8e0259d812766ea966c7/rpds_py-0.30.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:46e83c697b1f1c72b50e5ee5adb4353eef7406fb3f2043d64c33f20ad1c2fc53", size = 553371, upload-time = "2025-11-30T20:23:46.004Z" }, - { url = "https://files.pythonhosted.org/packages/72/c7/81dadd7b27c8ee391c132a6b192111ca58d866577ce2d9b0ca157552cce0/rpds_py-0.30.0-cp314-cp314-win32.whl", hash = "sha256:ee454b2a007d57363c2dfd5b6ca4a5d7e2c518938f8ed3b706e37e5d470801ed", size = 215298, upload-time = "2025-11-30T20:23:47.696Z" }, - { url = "https://files.pythonhosted.org/packages/3e/d2/1aaac33287e8cfb07aab2e6b8ac1deca62f6f65411344f1433c55e6f3eb8/rpds_py-0.30.0-cp314-cp314-win_amd64.whl", hash = "sha256:95f0802447ac2d10bcc69f6dc28fe95fdf17940367b21d34e34c737870758950", size = 228604, upload-time = "2025-11-30T20:23:49.501Z" }, - { url = "https://files.pythonhosted.org/packages/e8/95/ab005315818cc519ad074cb7784dae60d939163108bd2b394e60dc7b5461/rpds_py-0.30.0-cp314-cp314-win_arm64.whl", hash = "sha256:613aa4771c99f03346e54c3f038e4cc574ac09a3ddfb0e8878487335e96dead6", size = 222391, upload-time = "2025-11-30T20:23:50.96Z" }, - { url = "https://files.pythonhosted.org/packages/9e/68/154fe0194d83b973cdedcdcc88947a2752411165930182ae41d983dcefa6/rpds_py-0.30.0-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:7e6ecfcb62edfd632e56983964e6884851786443739dbfe3582947e87274f7cb", size = 364868, upload-time = "2025-11-30T20:23:52.494Z" }, - { url = "https://files.pythonhosted.org/packages/83/69/8bbc8b07ec854d92a8b75668c24d2abcb1719ebf890f5604c61c9369a16f/rpds_py-0.30.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:a1d0bc22a7cdc173fedebb73ef81e07faef93692b8c1ad3733b67e31e1b6e1b8", size = 353747, upload-time = "2025-11-30T20:23:54.036Z" }, - { url = "https://files.pythonhosted.org/packages/ab/00/ba2e50183dbd9abcce9497fa5149c62b4ff3e22d338a30d690f9af970561/rpds_py-0.30.0-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0d08f00679177226c4cb8c5265012eea897c8ca3b93f429e546600c971bcbae7", size = 383795, upload-time = "2025-11-30T20:23:55.556Z" }, - { url = "https://files.pythonhosted.org/packages/05/6f/86f0272b84926bcb0e4c972262f54223e8ecc556b3224d281e6598fc9268/rpds_py-0.30.0-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:5965af57d5848192c13534f90f9dd16464f3c37aaf166cc1da1cae1fd5a34898", size = 393330, upload-time = "2025-11-30T20:23:57.033Z" }, - { url = "https://files.pythonhosted.org/packages/cb/e9/0e02bb2e6dc63d212641da45df2b0bf29699d01715913e0d0f017ee29438/rpds_py-0.30.0-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:9a4e86e34e9ab6b667c27f3211ca48f73dba7cd3d90f8d5b11be56e5dbc3fb4e", size = 518194, upload-time = "2025-11-30T20:23:58.637Z" }, - { url = "https://files.pythonhosted.org/packages/ee/ca/be7bca14cf21513bdf9c0606aba17d1f389ea2b6987035eb4f62bd923f25/rpds_py-0.30.0-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:e5d3e6b26f2c785d65cc25ef1e5267ccbe1b069c5c21b8cc724efee290554419", size = 408340, upload-time = "2025-11-30T20:24:00.2Z" }, - { url = "https://files.pythonhosted.org/packages/c2/c7/736e00ebf39ed81d75544c0da6ef7b0998f8201b369acf842f9a90dc8fce/rpds_py-0.30.0-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:626a7433c34566535b6e56a1b39a7b17ba961e97ce3b80ec62e6f1312c025551", size = 383765, upload-time = "2025-11-30T20:24:01.759Z" }, - { url = "https://files.pythonhosted.org/packages/4a/3f/da50dfde9956aaf365c4adc9533b100008ed31aea635f2b8d7b627e25b49/rpds_py-0.30.0-cp314-cp314t-manylinux_2_31_riscv64.whl", hash = "sha256:acd7eb3f4471577b9b5a41baf02a978e8bdeb08b4b355273994f8b87032000a8", size = 396834, upload-time = "2025-11-30T20:24:03.687Z" }, - { url = "https://files.pythonhosted.org/packages/4e/00/34bcc2565b6020eab2623349efbdec810676ad571995911f1abdae62a3a0/rpds_py-0.30.0-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:fe5fa731a1fa8a0a56b0977413f8cacac1768dad38d16b3a296712709476fbd5", size = 415470, upload-time = "2025-11-30T20:24:05.232Z" }, - { url = "https://files.pythonhosted.org/packages/8c/28/882e72b5b3e6f718d5453bd4d0d9cf8df36fddeb4ddbbab17869d5868616/rpds_py-0.30.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:74a3243a411126362712ee1524dfc90c650a503502f135d54d1b352bd01f2404", size = 565630, upload-time = "2025-11-30T20:24:06.878Z" }, - { url = "https://files.pythonhosted.org/packages/3b/97/04a65539c17692de5b85c6e293520fd01317fd878ea1995f0367d4532fb1/rpds_py-0.30.0-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:3e8eeb0544f2eb0d2581774be4c3410356eba189529a6b3e36bbbf9696175856", size = 591148, upload-time = "2025-11-30T20:24:08.445Z" }, - { url = "https://files.pythonhosted.org/packages/85/70/92482ccffb96f5441aab93e26c4d66489eb599efdcf96fad90c14bbfb976/rpds_py-0.30.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:dbd936cde57abfee19ab3213cf9c26be06d60750e60a8e4dd85d1ab12c8b1f40", size = 556030, upload-time = "2025-11-30T20:24:10.956Z" }, - { url = "https://files.pythonhosted.org/packages/20/53/7c7e784abfa500a2b6b583b147ee4bb5a2b3747a9166bab52fec4b5b5e7d/rpds_py-0.30.0-cp314-cp314t-win32.whl", hash = "sha256:dc824125c72246d924f7f796b4f63c1e9dc810c7d9e2355864b3c3a73d59ade0", size = 211570, upload-time = "2025-11-30T20:24:12.735Z" }, - { url = "https://files.pythonhosted.org/packages/d0/02/fa464cdfbe6b26e0600b62c528b72d8608f5cc49f96b8d6e38c95d60c676/rpds_py-0.30.0-cp314-cp314t-win_amd64.whl", hash = "sha256:27f4b0e92de5bfbc6f86e43959e6edd1425c33b5e69aab0984a72047f2bcf1e3", size = 226532, upload-time = "2025-11-30T20:24:14.634Z" }, - { url = "https://files.pythonhosted.org/packages/69/71/3f34339ee70521864411f8b6992e7ab13ac30d8e4e3309e07c7361767d91/rpds_py-0.30.0-pp311-pypy311_pp73-macosx_10_12_x86_64.whl", hash = "sha256:c2262bdba0ad4fc6fb5545660673925c2d2a5d9e2e0fb603aad545427be0fc58", size = 372292, upload-time = "2025-11-30T20:24:16.537Z" }, - { url = "https://files.pythonhosted.org/packages/57/09/f183df9b8f2d66720d2ef71075c59f7e1b336bec7ee4c48f0a2b06857653/rpds_py-0.30.0-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:ee6af14263f25eedc3bb918a3c04245106a42dfd4f5c2285ea6f997b1fc3f89a", size = 362128, upload-time = "2025-11-30T20:24:18.086Z" }, - { url = "https://files.pythonhosted.org/packages/7a/68/5c2594e937253457342e078f0cc1ded3dd7b2ad59afdbf2d354869110a02/rpds_py-0.30.0-pp311-pypy311_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:3adbb8179ce342d235c31ab8ec511e66c73faa27a47e076ccc92421add53e2bb", size = 391542, upload-time = "2025-11-30T20:24:20.092Z" }, - { url = "https://files.pythonhosted.org/packages/49/5c/31ef1afd70b4b4fbdb2800249f34c57c64beb687495b10aec0365f53dfc4/rpds_py-0.30.0-pp311-pypy311_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:250fa00e9543ac9b97ac258bd37367ff5256666122c2d0f2bc97577c60a1818c", size = 404004, upload-time = "2025-11-30T20:24:22.231Z" }, - { url = "https://files.pythonhosted.org/packages/e3/63/0cfbea38d05756f3440ce6534d51a491d26176ac045e2707adc99bb6e60a/rpds_py-0.30.0-pp311-pypy311_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:9854cf4f488b3d57b9aaeb105f06d78e5529d3145b1e4a41750167e8c213c6d3", size = 527063, upload-time = "2025-11-30T20:24:24.302Z" }, - { url = "https://files.pythonhosted.org/packages/42/e6/01e1f72a2456678b0f618fc9a1a13f882061690893c192fcad9f2926553a/rpds_py-0.30.0-pp311-pypy311_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:993914b8e560023bc0a8bf742c5f303551992dcb85e247b1e5c7f4a7d145bda5", size = 413099, upload-time = "2025-11-30T20:24:25.916Z" }, - { url = "https://files.pythonhosted.org/packages/b8/25/8df56677f209003dcbb180765520c544525e3ef21ea72279c98b9aa7c7fb/rpds_py-0.30.0-pp311-pypy311_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:58edca431fb9b29950807e301826586e5bbf24163677732429770a697ffe6738", size = 392177, upload-time = "2025-11-30T20:24:27.834Z" }, - { url = "https://files.pythonhosted.org/packages/4a/b4/0a771378c5f16f8115f796d1f437950158679bcd2a7c68cf251cfb00ed5b/rpds_py-0.30.0-pp311-pypy311_pp73-manylinux_2_31_riscv64.whl", hash = "sha256:dea5b552272a944763b34394d04577cf0f9bd013207bc32323b5a89a53cf9c2f", size = 406015, upload-time = "2025-11-30T20:24:29.457Z" }, - { url = "https://files.pythonhosted.org/packages/36/d8/456dbba0af75049dc6f63ff295a2f92766b9d521fa00de67a2bd6427d57a/rpds_py-0.30.0-pp311-pypy311_pp73-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:ba3af48635eb83d03f6c9735dfb21785303e73d22ad03d489e88adae6eab8877", size = 423736, upload-time = "2025-11-30T20:24:31.22Z" }, - { url = "https://files.pythonhosted.org/packages/13/64/b4d76f227d5c45a7e0b796c674fd81b0a6c4fbd48dc29271857d8219571c/rpds_py-0.30.0-pp311-pypy311_pp73-musllinux_1_2_aarch64.whl", hash = "sha256:dff13836529b921e22f15cb099751209a60009731a68519630a24d61f0b1b30a", size = 573981, upload-time = "2025-11-30T20:24:32.934Z" }, - { url = "https://files.pythonhosted.org/packages/20/91/092bacadeda3edf92bf743cc96a7be133e13a39cdbfd7b5082e7ab638406/rpds_py-0.30.0-pp311-pypy311_pp73-musllinux_1_2_i686.whl", hash = "sha256:1b151685b23929ab7beec71080a8889d4d6d9fa9a983d213f07121205d48e2c4", size = 599782, upload-time = "2025-11-30T20:24:35.169Z" }, - { url = "https://files.pythonhosted.org/packages/d1/b7/b95708304cd49b7b6f82fdd039f1748b66ec2b21d6a45180910802f1abf1/rpds_py-0.30.0-pp311-pypy311_pp73-musllinux_1_2_x86_64.whl", hash = "sha256:ac37f9f516c51e5753f27dfdef11a88330f04de2d564be3991384b2f3535d02e", size = 562191, upload-time = "2025-11-30T20:24:36.853Z" }, -] - -[[package]] -name = "rpds-py" -version = "2026.6.3" -source = { registry = "https://pypi.org/simple" } -resolution-markers = [ - "python_full_version >= '3.14'", - "python_full_version >= '3.11' and python_full_version < '3.14'", -] -sdist = { url = "https://files.pythonhosted.org/packages/aa/2a/9618a122aeb2a169a28b03889a2995fe297588964333d4a7d67bdf46e147/rpds_py-2026.6.3.tar.gz", hash = "sha256:1cebd1337c242e4ec2293e541f712b2da849b29f48f0c293684b71c0632625d4", size = 64051, upload-time = "2026-06-30T07:17:53.009Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/94/1f/a2dca5ffdbf1d475ffc4e80e4d5d720ff3a00f691795910116960ee12511/rpds_py-2026.6.3-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:7b689145a1485c335569bd056464f3243a29af7ed3871c7be31ad624ba239bc7", size = 342174, upload-time = "2026-06-30T07:14:54.821Z" }, - { url = "https://files.pythonhosted.org/packages/4d/dc/323d08583c0832911768663d1944f0107fcd4088704858d84b5e06d105a0/rpds_py-2026.6.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:db08f45aecde626498fb3df07bcf6d2ec040af42e859a4f5040d79c200342911", size = 345513, upload-time = "2026-06-30T07:14:56.515Z" }, - { url = "https://files.pythonhosted.org/packages/0b/2a/e31989834d18d2f26ec1d2774c5b1eb3331df4ea8ada525175294c94b48a/rpds_py-2026.6.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:acc992ab27b15f852c76755eb2ab7dce86585ddadba6fa5946e58556088845b4", size = 373783, upload-time = "2026-06-30T07:14:57.736Z" }, - { url = "https://files.pythonhosted.org/packages/87/fe/e80107ee3639585c9941c17d6a42cd65325022f656c023191fce78c324c8/rpds_py-2026.6.3-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:7f88d653e7b3b779d71ae7454e20dcc9b6bae903f33c269db9f2be41bda3f261", size = 378316, upload-time = "2026-06-30T07:14:59.077Z" }, - { url = "https://files.pythonhosted.org/packages/22/6f/81e3adf81acfb6fa694de2a6e4e7d8863121e3e0799e0a7725e6cf5679c4/rpds_py-2026.6.3-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:e52655eaf81e32593abedaa4bfe33170c8cfedf3365ed9be6e11e07f148f0278", size = 499423, upload-time = "2026-06-30T07:15:00.488Z" }, - { url = "https://files.pythonhosted.org/packages/2d/9a/41263969df0ce3d9af2a96d5005a288200af1989aed3354bfceb5fc0b21f/rpds_py-2026.6.3-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:dfcc8b909769d19db55c7cc9541eb64b9b774b1057ffffb4f1048070475bb9f9", size = 386077, upload-time = "2026-06-30T07:15:01.911Z" }, - { url = "https://files.pythonhosted.org/packages/5e/19/7e98f468bd50346faff5b10e5297374b443bfdddacc8e9fbc65984539597/rpds_py-2026.6.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9c1255b302953c86a486b81d330d5ee1d5bd937691ce271b6be0ef0e299eaab7", size = 371315, upload-time = "2026-06-30T07:15:03.317Z" }, - { url = "https://files.pythonhosted.org/packages/99/3c/2b973b4d371906a134b03decfea7f5d9835a2c6d263454392e15b64b5b18/rpds_py-2026.6.3-cp311-cp311-manylinux_2_31_riscv64.whl", hash = "sha256:8d2294a31386bfa251d8c8a39472beee17db67d4f1a6eabea665d35c9a4461c3", size = 383502, upload-time = "2026-06-30T07:15:04.627Z" }, - { url = "https://files.pythonhosted.org/packages/98/2a/12e2799500af0a307bca76b63361c51f9fe479223561489c29eea1f2ee41/rpds_py-2026.6.3-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:f8f23ead891a3b762f35ab3b04623da7056545b48aa60d59957e6789914545da", size = 402673, upload-time = "2026-06-30T07:15:05.856Z" }, - { url = "https://files.pythonhosted.org/packages/2d/e3/21e5872d165fe08be4f229e3d5ee9d90019c0bf0e5538de60dbd54009450/rpds_py-2026.6.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:421aba32367055614287a4292b6a17f1939c9452299f7a0209c117e990b646d4", size = 549964, upload-time = "2026-06-30T07:15:07.159Z" }, - { url = "https://files.pythonhosted.org/packages/1a/d0/5ee0fe36844297de8123bee27bc12078c1a7416ad9f1b8a8ca18d6b0c0ac/rpds_py-2026.6.3-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:1e5822dfc2f0d4ab7e745eaa6d85945069329beeccef965af3f3bb26058fcab6", size = 615446, upload-time = "2026-06-30T07:15:08.531Z" }, - { url = "https://files.pythonhosted.org/packages/b1/80/1ea5873cb683f2fbe5f21b23ea1f6d179ead19f3c5b249b7eb5dca568ef2/rpds_py-2026.6.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:83e35b57523816c8613fd0776b40cd8bb9f596b37ddd2692eb4a6bb5ab2f8c93", size = 576975, upload-time = "2026-06-30T07:15:09.97Z" }, - { url = "https://files.pythonhosted.org/packages/c9/e1/90ef639217a5ddb15b7f4f61b1c33911fd044ad03c311bafdd2bcab85582/rpds_py-2026.6.3-cp311-cp311-win32.whl", hash = "sha256:de3eceba0b683bcbb1ab93da016d0270df1f9ae7be716b40214c5dafac6ea45a", size = 204453, upload-time = "2026-06-30T07:15:11.324Z" }, - { url = "https://files.pythonhosted.org/packages/f2/b7/b7a1695d7af36f521fb11e80d6d3adbd744f73b921859bd3c2a2c0dc706f/rpds_py-2026.6.3-cp311-cp311-win_amd64.whl", hash = "sha256:2c54a076ca4d370980ab57bc0e31df57bbe8d41340436a90ef8b1219a3cbb127", size = 223219, upload-time = "2026-06-30T07:15:12.476Z" }, - { url = "https://files.pythonhosted.org/packages/d7/a2/145afacf796e4506062825941176ad9445c2dcf2b3b6a1f13d3030a15e19/rpds_py-2026.6.3-cp311-cp311-win_arm64.whl", hash = "sha256:168c733a7112e071bb7a66460e667edfcff06c017a3c523f7a8a8e08d0140804", size = 219137, upload-time = "2026-06-30T07:15:13.631Z" }, - { url = "https://files.pythonhosted.org/packages/5c/be/2e8974163072e7bab7df1a5acd54c4498e75e35d6d18b864d3a9d5dadc92/rpds_py-2026.6.3-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:a0811d33247c3d6128a3001d763f2aa056bb3425204335400ac54f89eec3a0d0", size = 343691, upload-time = "2026-06-30T07:15:14.96Z" }, - { url = "https://files.pythonhosted.org/packages/a4/73/319dfa745dd668efe89309141ded489126461fcecd2b8f3a3cda185129b6/rpds_py-2026.6.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:538949e262e46caa31ac01bdb3c1e8f642622922cacbabbae6a8445d9dc33eaf", size = 338542, upload-time = "2026-06-30T07:15:16.267Z" }, - { url = "https://files.pythonhosted.org/packages/21/63/4239893be1c4d09b709b1a8f6be4188f0870084ff547f46606b8a75f1b03/rpds_py-2026.6.3-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:55927d532399c2c646100ff7feb48eaa940ad70f42cd68e1328f3ded9f81ca24", size = 368180, upload-time = "2026-06-30T07:15:17.62Z" }, - { url = "https://files.pythonhosted.org/packages/1c/ca/9c5de382225234ceb37b1844ebdb140db12b2a278bb9efe2fcd19f6c82ce/rpds_py-2026.6.3-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:f56f1695bc5c0871cbc33dc0130fcf503aab0c57dcc5a6700a4f49eba4f2652e", size = 375067, upload-time = "2026-06-30T07:15:18.952Z" }, - { url = "https://files.pythonhosted.org/packages/87/dc/863f69d1bf04ade34b7fe0d59b9fdf6f0135fe2d7cbca74f1d665589559d/rpds_py-2026.6.3-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:270b293dae9058fc9fcedab50f13cebf46fb8ed1d1d54e0521a9da5d6b211975", size = 490509, upload-time = "2026-06-30T07:15:20.434Z" }, - { url = "https://files.pythonhosted.org/packages/ce/ef/eac16a12048b45ec7c7fa94f2be3438a5f26bf9cc8580b18a1cfd609b7f6/rpds_py-2026.6.3-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:127565fead0a10943b282957bd5447804ff3160ad79f2ad2635e6d249e380680", size = 382754, upload-time = "2026-06-30T07:15:21.831Z" }, - { url = "https://files.pythonhosted.org/packages/04/8f/d2f3f532616be4d06c316ef119683e832bd3d41e112bf3a88f4151c95b17/rpds_py-2026.6.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ecabd69db66de867690f9797f2f8fa27ba501bbc24540cbdbdc649cd15888ba6", size = 366189, upload-time = "2026-06-30T07:15:23.371Z" }, - { url = "https://files.pythonhosted.org/packages/e3/29/41a7b0e98a4b44cd676ab7598419623373eb43b20be68c084935c1a8cf88/rpds_py-2026.6.3-cp312-cp312-manylinux_2_31_riscv64.whl", hash = "sha256:58eadac9cd119677b60e1cf8ac4052f35949d71b8a9e5556efccbe82533cf22a", size = 377750, upload-time = "2026-06-30T07:15:24.659Z" }, - { url = "https://files.pythonhosted.org/packages/2e/05/ecda0bec46f9a1565090bcdc941d023f6a25aff85fda28f89f8d19878152/rpds_py-2026.6.3-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:7491ee23305ac3eb59e492b6945881f5cd77a6f731061a3f25b77fd40f9e99a4", size = 395576, upload-time = "2026-06-30T07:15:25.987Z" }, - { url = "https://files.pythonhosted.org/packages/68/a8/6ed52f03ee6cb854ce78785cc9a9a672eb880e83fd7224d471f667d151f1/rpds_py-2026.6.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:2c99f7e8ccb3dd6e3e4bfeac657a7b208c9bac8075f4b078c02d7404c34107fa", size = 543807, upload-time = "2026-06-30T07:15:27.356Z" }, - { url = "https://files.pythonhosted.org/packages/8f/d6/156c0d3eea27ba09b92562ba2364ba124c0a061b199e17eac637cd25a5e2/rpds_py-2026.6.3-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:62698275682bf121181861295c9181e789030a2d516071f5b8f3c23c170cd0fc", size = 611187, upload-time = "2026-06-30T07:15:28.931Z" }, - { url = "https://files.pythonhosted.org/packages/f1/31/774212ed989c62f7f310220089f9b0a3fb8f40f5443d1727abd5d9f52bc9/rpds_py-2026.6.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:a214c993455f99a89aaeadc9b21241900037adc9d97203e374d75513c5911822", size = 573030, upload-time = "2026-06-30T07:15:30.553Z" }, - { url = "https://files.pythonhosted.org/packages/c9/50/22f73127a41f1ce4f87fe39aadfb9a126345801c274aa93ae88456249327/rpds_py-2026.6.3-cp312-cp312-win32.whl", hash = "sha256:501f9f04a588d6a09179368c57071301445191767c64e4b52a6aa9871f1ef5ed", size = 202185, upload-time = "2026-06-30T07:15:32.027Z" }, - { url = "https://files.pythonhosted.org/packages/04/3a/f0ee4d4dde9d3b69dedf1b5f74e7a40017046d55052d173e418c6a94f960/rpds_py-2026.6.3-cp312-cp312-win_amd64.whl", hash = "sha256:2c958bf94822e9290a40aaf2a822d4bc5c88099093e3948ad6c571eca9272e5f", size = 220394, upload-time = "2026-06-30T07:15:33.359Z" }, - { url = "https://files.pythonhosted.org/packages/f3/83/3382fe37f809b59f02aac04dbc4e765b480b46ee0227ed516e3bdc4d3dfc/rpds_py-2026.6.3-cp312-cp312-win_arm64.whl", hash = "sha256:22bffe6042b9bcb0822bcd1955ec00e245daf17b4344e4ed8e9551b976b63e96", size = 215753, upload-time = "2026-06-30T07:15:34.778Z" }, - { url = "https://files.pythonhosted.org/packages/a4/9e/b818ee580026ec578138e961027a68820c40afeb1ec8f6819b54fb99e196/rpds_py-2026.6.3-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:3cfe765c1da0072636ca06628261e0ea05688e160d5c8a03e0217c3854037223", size = 343012, upload-time = "2026-06-30T07:15:36.005Z" }, - { url = "https://files.pythonhosted.org/packages/f3/6b/686d9dc4359a8f163cfbbf89ee0b4e586431de22fe8248edb63a8cf50d49/rpds_py-2026.6.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:f4d78253f6996be4901669ad25319f842f740eccf4d58e3c7f3dd39e6dde1d8f", size = 338203, upload-time = "2026-06-30T07:15:37.462Z" }, - { url = "https://files.pythonhosted.org/packages/9e/9b/069aa329940f8207615e091f5eedbbd40e1e15eac68a0790fd05ccdf796c/rpds_py-2026.6.3-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:54f45a148e28767bf343d33a684693c70e451c6f4c0e9904709a723fafbdfc1f", size = 367984, upload-time = "2026-06-30T07:15:39.008Z" }, - { url = "https://files.pythonhosted.org/packages/14/db/34c203e4becff3703e4d3bc121842c00b8689197f398161203a880052f4e/rpds_py-2026.6.3-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:842e7b070435622248c7a2c44ae53fa1440e073cc3023bc919fed570884097a7", size = 374815, upload-time = "2026-06-30T07:15:40.253Z" }, - { url = "https://files.pythonhosted.org/packages/ee/7d/8071067d2cc453d916ad836e828c943f575e8a44612537759002a1e07381/rpds_py-2026.6.3-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:8020133a74bd81b4572dd8e4be028a6b1ebcd70e6726edc3918008c08bee6ee6", size = 490545, upload-time = "2026-06-30T07:15:41.729Z" }, - { url = "https://files.pythonhosted.org/packages/a3/42/da06c5aa8f0484ff07f270787434204d9f4535e2f8c3b51ed402267e63c3/rpds_py-2026.6.3-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:cdc7e35386f3847df728fbcb5e887e2d79c19e2fa1eba9e51b6621d23e3243af", size = 382828, upload-time = "2026-06-30T07:15:43.327Z" }, - { url = "https://files.pythonhosted.org/packages/57/d7/fe978efc2ae50abe48eb7464668ea99f53c010c60aeebb7b35ad27f23661/rpds_py-2026.6.3-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:acac386b453c2516111b50985d60ce46e7fadb5ea71ae7b25f4c946935bf27cf", size = 365678, upload-time = "2026-06-30T07:15:44.992Z" }, - { url = "https://files.pythonhosted.org/packages/69/9d/1d8922e1990b2a6eb532b6ff53d3e73d2b3bbffc84116c75826bee73dfc6/rpds_py-2026.6.3-cp313-cp313-manylinux_2_31_riscv64.whl", hash = "sha256:425560c6fa0415f27261727bb20bd097568485e5eb0c121f1949417d1c516885", size = 377811, upload-time = "2026-06-30T07:15:46.523Z" }, - { url = "https://files.pythonhosted.org/packages/b1/3d/198dceafb4fb034a6a47347e1b0735d34e0bd4a50be4e898d408ee66cb14/rpds_py-2026.6.3-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:a550fb4950a06dde3beb4721f5ad4b25bf4513784665b0a8522c792e2bd822a4", size = 395382, upload-time = "2026-06-30T07:15:47.955Z" }, - { url = "https://files.pythonhosted.org/packages/1f/f1/13968e49655d40b6b19d8b9140296bbc6f1d86b3f0f6c346cf9f1adddf4b/rpds_py-2026.6.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:4f4bca01b63096f606e095734dd56e74e175f94cfbf24ff3d63281cec61f7bb7", size = 543832, upload-time = "2026-06-30T07:15:49.33Z" }, - { url = "https://files.pythonhosted.org/packages/ac/ab/289bcb1b90bd3e40a2900c561fa0e2087345ecbb094f0b870f2345142b7c/rpds_py-2026.6.3-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:ccffae9a092a00deb7efd545fe5e2c33c33b88e7c054337e9a74c179347d0b7d", size = 611011, upload-time = "2026-06-30T07:15:50.847Z" }, - { url = "https://files.pythonhosted.org/packages/1e/16/5043105e679436ccfbc8e5e0dd2d663ed18a8b8113515fd06a5e5d77c83e/rpds_py-2026.6.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:1cf01971c4f2c5553b772a542e4aaf191789cd331bc2cd4ff0e6e65ba49e1e97", size = 572431, upload-time = "2026-06-30T07:15:52.394Z" }, - { url = "https://files.pythonhosted.org/packages/85/ed/adab103321c0a6565d5ae1c2998349bc3ee175b82ccc5ae8fc04cc413075/rpds_py-2026.6.3-cp313-cp313-win32.whl", hash = "sha256:8c3d1e9c15b9d51ca0391e13da1a25a0a4df3c58a37c9dc368e0736cf7f69df0", size = 201710, upload-time = "2026-06-30T07:15:53.894Z" }, - { url = "https://files.pythonhosted.org/packages/7b/ed/a03b09668e74e5dabbf2e211f6468e1820c0552f7b0500082da31841bf7b/rpds_py-2026.6.3-cp313-cp313-win_amd64.whl", hash = "sha256:9250a9a0a6fd4648b3f868da8d91a4c52b5811a62df58e753d50ae4454a36f80", size = 219454, upload-time = "2026-06-30T07:15:55.25Z" }, - { url = "https://files.pythonhosted.org/packages/27/17/b8642c12930b71bc2b25831f6708ccf0f75abcd11883932ec9ce54ba3a78/rpds_py-2026.6.3-cp313-cp313-win_arm64.whl", hash = "sha256:900a67df3fd1660b035a4761c4ce73c382ea6b35f90f9863c36c6fd8bf8b09bb", size = 215063, upload-time = "2026-06-30T07:15:56.573Z" }, - { url = "https://files.pythonhosted.org/packages/b6/36/7fbe9dcdaf857fb3f63c2a2284b62492d95f5e8334e947e5fb6e7f68c9be/rpds_py-2026.6.3-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:931908d9fc855d8f74783377822be318edb6dcb19e47169dc038f9a1bf60b06e", size = 344510, upload-time = "2026-06-30T07:15:57.921Z" }, - { url = "https://files.pythonhosted.org/packages/ba/54/f785cc3d3f60839ca57a5af4927a9f347b07b2799c373fc20f7949f87c7e/rpds_py-2026.6.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:d7469697dce35be237db177d42e2a2ee26e6dcc5fc052078a6fefabd288c6edd", size = 339495, upload-time = "2026-06-30T07:15:59.238Z" }, - { url = "https://files.pythonhosted.org/packages/63/ef/d4cdaf309e6b095b43597103cf8c0b951d6cca2acce68c474f75ec12e0c7/rpds_py-2026.6.3-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:bcfbcf66006befb9fd2aeaa9e01feaf881b4dc330a02ba07d2322b1c11be7b5d", size = 369454, upload-time = "2026-06-30T07:16:01.021Z" }, - { url = "https://files.pythonhosted.org/packages/96/4a/9559a68b7ee15db09d7981212e8c2e219d2a1d6d4faa0391d813c3496a36/rpds_py-2026.6.3-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:847927daf4cffbd4e90e42bc890069897101edd015f956cb8721b3473372edda", size = 374583, upload-time = "2026-06-30T07:16:02.287Z" }, - { url = "https://files.pythonhosted.org/packages/ef/75/8964aa7d2c6e8ac43eba8eb6e6b0fdda1f46d39f2fc3e6aa9f2cb17f485d/rpds_py-2026.6.3-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:aca6c1ef08a82bfe327cc156da694660f599923e2e6665b6d81c9c2d0ac9ffc8", size = 492919, upload-time = "2026-06-30T07:16:03.723Z" }, - { url = "https://files.pythonhosted.org/packages/8f/97/6908094ac804115e65aedfd90f1b5fee4eebebd3f6c4cfc5419939267565/rpds_py-2026.6.3-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:ae50181a047c871561212bb97f7932a2d45fb53e947bd9b57ebad85b529cbc53", size = 383725, upload-time = "2026-06-30T07:16:05.305Z" }, - { url = "https://files.pythonhosted.org/packages/d1/9c/0d1fdc2e7aba23e290d603bc494e97bd205bae262ce33c6b32a69768ed5e/rpds_py-2026.6.3-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:dc319e5a1de4b6913aac94bf6a2f9e847371e0a140a43dd4991db1a09bc2d504", size = 367255, upload-time = "2026-06-30T07:16:07.086Z" }, - { url = "https://files.pythonhosted.org/packages/c4/fe/f0209ca4a9ed074bc8acb44dfd0e81c3122e94c9689f5645b7973a866719/rpds_py-2026.6.3-cp314-cp314-manylinux_2_31_riscv64.whl", hash = "sha256:e4316bf32babbed84e691e352faf967ce2f0f024174a8643c37c94a1080374fc", size = 379060, upload-time = "2026-06-30T07:16:08.525Z" }, - { url = "https://files.pythonhosted.org/packages/c6/8d/f1cc54c616b9d8897de8738aac148d20afca93f68187475fe194d09a71b9/rpds_py-2026.6.3-cp314-cp314-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:8c6e5a2f750cc71c3e3b11d71661f21d6f9bc6cebc6564b1466417a1ec03ec77", size = 395960, upload-time = "2026-06-30T07:16:09.989Z" }, - { url = "https://files.pythonhosted.org/packages/fb/04/aafff00f73aeca2945f734f1d483c64ab8f472d0864ab02377fd8e89c3b2/rpds_py-2026.6.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:4470ce197d4090875cf6affbf1f853338387428df97c4fb7b7106317b8214698", size = 545356, upload-time = "2026-06-30T07:16:11.816Z" }, - { url = "https://files.pythonhosted.org/packages/fd/cc/e229663b9e4ddac5a4acbe9085dd80a71af2a5d356b8b39d6bff233f24b0/rpds_py-2026.6.3-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:ea964164cc9afa72d4d9b23cc28dafae93693c0a53e0b42acbff15b22c3f9ddd", size = 612319, upload-time = "2026-06-30T07:16:13.586Z" }, - { url = "https://files.pythonhosted.org/packages/e3/7a/8a0e6d3e6cd066af108b71b43122c3fe158dd9eb86acac626593a2582eb1/rpds_py-2026.6.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:639c8929aa0afe81be836b04de888460d6bed38b9c54cfc18da8f6bfabf5af5d", size = 573508, upload-time = "2026-06-30T07:16:15.23Z" }, - { url = "https://files.pythonhosted.org/packages/87/03/2a69ab618a789cf6cf85c86bb844c62d090e700ab1a2aa676b3741b6c516/rpds_py-2026.6.3-cp314-cp314-win32.whl", hash = "sha256:882076c00c0a608b131187055ddc5ae29f2e7eaf870d6168980420d58528a5c8", size = 202504, upload-time = "2026-06-30T07:16:16.893Z" }, - { url = "https://files.pythonhosted.org/packages/85/62/a3892ba945f4e24c78f352e5de3c7620d8479f73f211406a97263d13c7d2/rpds_py-2026.6.3-cp314-cp314-win_amd64.whl", hash = "sha256:0be972be84cfcaf46c8c6edf690ca0f154ac17babf1f6a955a51579b34ad2dc5", size = 220380, upload-time = "2026-06-30T07:16:18.108Z" }, - { url = "https://files.pythonhosted.org/packages/3d/e7/c2bd44dc831931815ad11ebb5f430b5a0a4d3caa9de837107876c30c3432/rpds_py-2026.6.3-cp314-cp314-win_arm64.whl", hash = "sha256:2a9c6f195058cb45335e8cc3802745c603d716eb96bc9625950c1aac71c0c703", size = 215976, upload-time = "2026-06-30T07:16:19.654Z" }, - { url = "https://files.pythonhosted.org/packages/79/9c/fff7b74bce9a091ec9a012a03f9ff5f69364eaf9451060dfc4486da2ffdd/rpds_py-2026.6.3-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:f90938e92afda60266da758ee7d363447f7f0138c9559f9e1811629580582d90", size = 346840, upload-time = "2026-06-30T07:16:21.268Z" }, - { url = "https://files.pythonhosted.org/packages/e9/44/77bcb1168b33704908295533d27f10eb811e9e3e193e8993dc99572211d3/rpds_py-2026.6.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:ec829541c45bca16e61c7ae50c20501f213605beb75d1aba91a6ee37fbbb56a4", size = 340282, upload-time = "2026-06-30T07:16:22.875Z" }, - { url = "https://files.pythonhosted.org/packages/87/3c/7a9081c7c9e645b39efe19e4ffbeccd80add246327cd9b888aecffd72317/rpds_py-2026.6.3-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:afd70d95892096cdb26f15a00c45907b17817577aa8d1c76b2dcc2788391f9e9", size = 370403, upload-time = "2026-06-30T07:16:24.415Z" }, - { url = "https://files.pythonhosted.org/packages/f7/69/af47021eb7dad6ff3396cb001c08f0f3c4d06c20253f75be6421a59fe6b7/rpds_py-2026.6.3-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:29dfa0533a5d4c94d4dfa1b694fcb56c9c63aad8330ffdd816fd225d0a7a162f", size = 376055, upload-time = "2026-06-30T07:16:26.111Z" }, - { url = "https://files.pythonhosted.org/packages/81/fc/a3bcf517084396a6dd258c592567a3c011ba4557f2fde23dceaf26e74f2e/rpds_py-2026.6.3-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:af05d726809bff6b141be124d4c7ce998f9c9c7f30edb1f46c07aa103d540b41", size = 494419, upload-time = "2026-06-30T07:16:27.596Z" }, - { url = "https://files.pythonhosted.org/packages/c9/eb/13d529d1788135425c7bf207f8463458ca5d92e43f3f701365b83e9dffc1/rpds_py-2026.6.3-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:9826217f048f620d9a712672818bf231442c1b35d96b227a07eabd11b4bb6945", size = 384848, upload-time = "2026-06-30T07:16:29.183Z" }, - { url = "https://files.pythonhosted.org/packages/8e/f4/b7ac49f30013aba8f7b9566b1dd07e81de95e708c1374b7bacc5b9bc5c9c/rpds_py-2026.6.3-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:536bceea4fa4acf7e1c61da2b5786304367c816c8895be71b8f537c480b0ea1f", size = 371369, upload-time = "2026-06-30T07:16:30.912Z" }, - { url = "https://files.pythonhosted.org/packages/31/86/6260bafa622f788b07ddec0e52d810305c8b9b0b8c27f58a2ab04bf62b4f/rpds_py-2026.6.3-cp314-cp314t-manylinux_2_31_riscv64.whl", hash = "sha256:bc0011654b91cc4fb2ae701bec0a0ba1e552c0714247fa7af6c59e0ccfa3a4e1", size = 379673, upload-time = "2026-06-30T07:16:32.486Z" }, - { url = "https://files.pythonhosted.org/packages/19/c3/03f1ee79a047b48daeca157c89a18509cde22b6b951d642b9b0af1be660a/rpds_py-2026.6.3-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:539d75de9e0d536c84ff18dfeb805398e58227001ce09231a26a08b9aed1ee0e", size = 397500, upload-time = "2026-06-30T07:16:34.471Z" }, - { url = "https://files.pythonhosted.org/packages/f0/95/8ed0cd8c377dca12aea498f119fe639fc474d1461545c39d2b5872eb1c0f/rpds_py-2026.6.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:166cf54d9f44fc6ceb53c7860258dde44a81406646de79f8ed3234fca3b6e538", size = 545978, upload-time = "2026-06-30T07:16:36.45Z" }, - { url = "https://files.pythonhosted.org/packages/d3/f2/0eb57f0eaa83f8fc152a7e03de968ab77e1f00732bebc892b190c6eebde7/rpds_py-2026.6.3-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:d34c20167764fbcf927194d532dd7e0c56772f0a5f943fa5ef9e9afbba8fb9db", size = 613350, upload-time = "2026-06-30T07:16:38.213Z" }, - { url = "https://files.pythonhosted.org/packages/5b/de/e0674bdbc3ef7634989b3f854c3f34bc1f587d36e5bfdc5c378d57034619/rpds_py-2026.6.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:ea7bb13b7c9a29791f87a0387ba7d3ad3a6d783d827e4d3f27b40a0ff44495e2", size = 576486, upload-time = "2026-06-30T07:16:39.797Z" }, - { url = "https://files.pythonhosted.org/packages/f2/f6/21101359743cd136ada781e8210a85769578422ba460672eea0e29739200/rpds_py-2026.6.3-cp314-cp314t-win32.whl", hash = "sha256:6de4744d05bd1aa1be4ed7ea1189e3979196808008113bbbf899a460966b925e", size = 201068, upload-time = "2026-06-30T07:16:41.316Z" }, - { url = "https://files.pythonhosted.org/packages/a6/b2/9574d4d44f7760c2aa32d92a0a4f41698e33f5b204a0bf5c9758f52c79d5/rpds_py-2026.6.3-cp314-cp314t-win_amd64.whl", hash = "sha256:c7b9a2f8f4d8e90af72571d3d495deebdd7e3c75451f5b41719aee166e940fc2", size = 220600, upload-time = "2026-06-30T07:16:43.091Z" }, - { url = "https://files.pythonhosted.org/packages/08/ae/f23a2697e6ee6340a578b0f136be6483657bef0c6f9497b752bb5c0964bb/rpds_py-2026.6.3-cp315-cp315-macosx_10_12_x86_64.whl", hash = "sha256:e059c5dde6452b44424bd1834557556c226b57781dee1227af23518459722b13", size = 344726, upload-time = "2026-06-30T07:16:44.5Z" }, - { url = "https://files.pythonhosted.org/packages/c3/63/e7b3a1a5358dd32c930a1062d8e15b67fd6e8922e81df9e91706d66ee5c8/rpds_py-2026.6.3-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:2f7c26fbc5acd2522b95d4177fe4710ffd8e9b20529e703ffbf8db4d93903f05", size = 339587, upload-time = "2026-06-30T07:16:46.255Z" }, - { url = "https://files.pythonhosted.org/packages/ec/64/10a85681916ca55fffb91b0a211f84e34297c109243484dd6394660a8a7c/rpds_py-2026.6.3-cp315-cp315-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a3086b538543802f84c843911242db20447de00d8752dd0efc936dbcf02218ba", size = 369585, upload-time = "2026-06-30T07:16:48.101Z" }, - { url = "https://files.pythonhosted.org/packages/76/c2/baf95c7c38823e12ba34407c5f5767a89e5cf2233895e56f608167ae9493/rpds_py-2026.6.3-cp315-cp315-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:8f2e5c5ee828d42cb11760761c0af6507927bec42d0ad5458f97c9203b054617", size = 375479, upload-time = "2026-06-30T07:16:49.93Z" }, - { url = "https://files.pythonhosted.org/packages/6a/94/0aad06c72d65101e11d33528d438cda99a39ce0da99466e156158f2541d3/rpds_py-2026.6.3-cp315-cp315-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:ed0c1e5d10cdc7135537988c74a0188da68e2f3c30813ba3744ab1e42e0480f9", size = 492418, upload-time = "2026-06-30T07:16:51.641Z" }, - { url = "https://files.pythonhosted.org/packages/b5/17/de3f5a479a1f056535d7489819639d8cd591ea6281d700390b43b1abd745/rpds_py-2026.6.3-cp315-cp315-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:8c2642a7603ec0b16ed77da4555db3b4b472341904873788327c0b0d7b95f1bb", size = 384123, upload-time = "2026-06-30T07:16:53.622Z" }, - { url = "https://files.pythonhosted.org/packages/46/7d/bf09bd1b145bb2671c03e1e6d1ab8651858d90d8c7dfeadd85a37a934fd8/rpds_py-2026.6.3-cp315-cp315-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8e4320744c1ffdd95a603def63344bfab2d33edeab301c5007e7de9f9f5b3885", size = 367351, upload-time = "2026-06-30T07:16:55.241Z" }, - { url = "https://files.pythonhosted.org/packages/a3/ea/1bb734f314b8be319149ddee80b18bd41372bdcfbdf88d28131c0cd37719/rpds_py-2026.6.3-cp315-cp315-manylinux_2_31_riscv64.whl", hash = "sha256:a9f4645593036b81bbdb36b9c8e0ea0d1c3fee968c4d59db0344c14087ef143a", size = 378827, upload-time = "2026-06-30T07:16:56.841Z" }, - { url = "https://files.pythonhosted.org/packages/4b/93/d9611e5b25e26df9a3649813ed66193ace9347a7c7fc4ab7cf70e94851c0/rpds_py-2026.6.3-cp315-cp315-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:e55d236be29255554da47abe5c577637db7c24a02b8b46f0ca9524c855801868", size = 395966, upload-time = "2026-06-30T07:16:58.557Z" }, - { url = "https://files.pythonhosted.org/packages/c3/cb/99d77e16e5534ae1d90629bbe419ba6ee170833a6a85e3aa1cc41726fbbc/rpds_py-2026.6.3-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:24e9c5386e16669b674a69c156c8eeefcb578f3b3397b713b08e6d60f3c7b187", size = 545680, upload-time = "2026-06-30T07:17:00.164Z" }, - { url = "https://files.pythonhosted.org/packages/59/15/11a29755f790cef7a2f755e8e14f4f0c33f39489e1893a632a2eee59672b/rpds_py-2026.6.3-cp315-cp315-musllinux_1_2_i686.whl", hash = "sha256:c60924535c75f1566b6eb75b5c31a48a43fef04fa2d0d201acbad8a9969c6107", size = 611853, upload-time = "2026-06-30T07:17:01.962Z" }, - { url = "https://files.pythonhosted.org/packages/68/86/0c27547e21644da938fb530f7e1a8148dd24d02db07e7a5f2567a17ce710/rpds_py-2026.6.3-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:38a2fea2787428f811719ceb9114cb78964a3138838320c29ac39526c79c16ba", size = 573715, upload-time = "2026-06-30T07:17:03.693Z" }, - { url = "https://files.pythonhosted.org/packages/29/71/4d8fcf700931815594bce892255bbd973b94efaf0fc1932b0590df18d886/rpds_py-2026.6.3-cp315-cp315-win32.whl", hash = "sha256:d483fe17f01ad64b7bf7cc38fcefff1ca9fb83f8c2b2542b68f97ffe0611b369", size = 202864, upload-time = "2026-06-30T07:17:05.746Z" }, - { url = "https://files.pythonhosted.org/packages/eb/62/b577562de0edbb55b2be85ce5fd09c33e386b9b13eee09833af4240fd5c4/rpds_py-2026.6.3-cp315-cp315-win_amd64.whl", hash = "sha256:67e3a721ffc5d8d2210d3671872298c4a84e4b8035cfe42ffd7cde35d772b146", size = 220430, upload-time = "2026-06-30T07:17:07.471Z" }, - { url = "https://files.pythonhosted.org/packages/c8/95/d6d0b2509825141eef60669a5739eec88dbc6a48053d6c92993a5704defe/rpds_py-2026.6.3-cp315-cp315-win_arm64.whl", hash = "sha256:6e84adbcf4bf841aed8116a8264b9f50b4cb3e7bd89b516122e616ac56ca269e", size = 215877, upload-time = "2026-06-30T07:17:09.008Z" }, - { url = "https://files.pythonhosted.org/packages/b7/bf/f3ea278f0afd615c1d0f19cb69043a41526e2bb600c2b536eb192218eb27/rpds_py-2026.6.3-cp315-cp315t-macosx_10_12_x86_64.whl", hash = "sha256:ae6dd8f10bd17aad820876d24caec9efdafd80a318d16c0a48edb5e136902c6b", size = 346933, upload-time = "2026-06-30T07:17:10.762Z" }, - { url = "https://files.pythonhosted.org/packages/9d/29/9907bdf1c5346763cf10b7f6852aad86652168c259def904cbe0082c5864/rpds_py-2026.6.3-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:bdbd97738551fca3917c1bd7188bec1920bb520104f28e7e1007f9ceb17b7690", size = 340274, upload-time = "2026-06-30T07:17:12.266Z" }, - { url = "https://files.pythonhosted.org/packages/6f/2c/8e03767b5778ef25cebf74a7a91a2c3806f8eced4c92cb7406bbe060756d/rpds_py-2026.6.3-cp315-cp315t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:8b95977e7211527ab0ba576e286d023389fbeeb32a6b7b771665d333c60e5342", size = 370763, upload-time = "2026-06-30T07:17:14.107Z" }, - { url = "https://files.pythonhosted.org/packages/2e/e1/df2a7e1ba2efd796af26194250b8d42c821b46592311595162af9ef0528d/rpds_py-2026.6.3-cp315-cp315t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:d15fde0e6fb0d88a60d221204873743e5d9f0b7d29165e62cd86d0413ad74ba6", size = 376467, upload-time = "2026-06-30T07:17:15.76Z" }, - { url = "https://files.pythonhosted.org/packages/6b/de/8a0814d1946af29cb068fb259aa8622f856df1d0bab58429448726b537f5/rpds_py-2026.6.3-cp315-cp315t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:a136d453475ac0fcbda502ef1e6504bd28d6d904700915d278deeab0d00fe140", size = 496689, upload-time = "2026-06-30T07:17:17.308Z" }, - { url = "https://files.pythonhosted.org/packages/df/f3/f19e0c852ba13694f5a79f3b719331051573cb5693feacf8a88ffffc3a71/rpds_py-2026.6.3-cp315-cp315t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:f826877d462181e5eb1c26a0026b8d0cab05d99844ecb6d8bf3627a2ca0c0442", size = 385340, upload-time = "2026-06-30T07:17:18.928Z" }, - { url = "https://files.pythonhosted.org/packages/e2/ae/7ec3a9d2d4351f99e37bcb06b6b6f954512646bfdbf9742e1de727865daf/rpds_py-2026.6.3-cp315-cp315t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:79486287de1730dbaff3dbd124d0ca4d2ef7f9d29bf2544f1f93c09b5bcbbd12", size = 372179, upload-time = "2026-06-30T07:17:20.539Z" }, - { url = "https://files.pythonhosted.org/packages/d3/ac/9cee911dff2aaa9a5a8354f6610bf2e6a616de9197c5fff4f54f82585f1e/rpds_py-2026.6.3-cp315-cp315t-manylinux_2_31_riscv64.whl", hash = "sha256:808345f53cb952433ca2816f1604ff3515608a81784954f38d4452acfe8e61d5", size = 379993, upload-time = "2026-06-30T07:17:22.212Z" }, - { url = "https://files.pythonhosted.org/packages/83/6b/7c2a07ba88d1e9a936612f7a5d067467ed03d971d5a06f7d309dff044a7e/rpds_py-2026.6.3-cp315-cp315t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:1967debc37f64f2c4dc90a7f563aec558b471966e12adcac4e1c4240496b6ebf", size = 398909, upload-time = "2026-06-30T07:17:23.66Z" }, - { url = "https://files.pythonhosted.org/packages/97/0b/776ffcb66783637b0031f6d58d6fb55913c8b5abf00aeecd46bf933fb477/rpds_py-2026.6.3-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:f0840b5b17057f7fd918b76183a4b5a0635f43e14eb2ce60dce1d4ee4707ea00", size = 546584, upload-time = "2026-06-30T07:17:25.264Z" }, - { url = "https://files.pythonhosted.org/packages/55/33/ba3bc04d7092bd553c9b2b195624992d2cc4f3de1f380b7b93cbee67bd79/rpds_py-2026.6.3-cp315-cp315t-musllinux_1_2_i686.whl", hash = "sha256:faa679d19a6696fd54259ad321251ad77a13e70e03dd834daa762a44fb6196ef", size = 614357, upload-time = "2026-06-30T07:17:26.888Z" }, - { url = "https://files.pythonhosted.org/packages/8b/71/14edf065f04630b1a8472f7653cad03f6c478bcf95ea0e6aed55451e33ea/rpds_py-2026.6.3-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:23a439f31ccbeff1574e24889128821d1f7917470e830cf6544dced1c662262a", size = 576533, upload-time = "2026-06-30T07:17:28.546Z" }, - { url = "https://files.pythonhosted.org/packages/ba/76/65002b08596c389105720a8c0d22298b8dc25a4baf89b2ce431343c8b1de/rpds_py-2026.6.3-cp315-cp315t-win32.whl", hash = "sha256:913ca42ccad3f8cc6e292b587ae8ae49c8c823e5dce51a736252fc7c7cdfa577", size = 201204, upload-time = "2026-06-30T07:17:30.193Z" }, - { url = "https://files.pythonhosted.org/packages/8c/97/d855d6b3c322d1f27e26f5241c42016b56cf01377ea8ed348285f54652f0/rpds_py-2026.6.3-cp315-cp315t-win_amd64.whl", hash = "sha256:ae3d4fe8c0b9213624fdce7279d70e3b148b682ca20719ebd193a23ebfa47324", size = 220719, upload-time = "2026-06-30T07:17:31.788Z" }, - { url = "https://files.pythonhosted.org/packages/b4/9c/f0d19ac587fd0e4ab6b72cda355e9c5a6166b01ef7e064e437aef8eb9fef/rpds_py-2026.6.3-pp311-pypy311_pp73-macosx_10_12_x86_64.whl", hash = "sha256:4cf2d36a2357e4d07bb5a4f98801265327b48256867816cfd2ceb001e9754a8f", size = 349791, upload-time = "2026-06-30T07:17:33.315Z" }, - { url = "https://files.pythonhosted.org/packages/38/c7/1d49d204c9fd2ee6c537601dc4c1ba921e03363ca576bfab94a00254ac9a/rpds_py-2026.6.3-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:30c6dc199b24a5e3e81d50da0f00858c5bbdb2617a750395687f4339c5818171", size = 352842, upload-time = "2026-06-30T07:17:34.897Z" }, - { url = "https://files.pythonhosted.org/packages/ac/e5/c0b5dc93cd0d4c06ce1f438907649514e2ea077bcd911e3154a51e96c38e/rpds_py-2026.6.3-pp311-pypy311_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9891e594296ab9dada6551c8e7b387b2721f27a67eecd528412e8906247a7b90", size = 382094, upload-time = "2026-06-30T07:17:36.514Z" }, - { url = "https://files.pythonhosted.org/packages/0d/54/ec0e907b4ca8d541112db352409bd15f871c9b243e0c92c9b5a46ae96f01/rpds_py-2026.6.3-pp311-pypy311_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:b5c2dc92304aa48a4a60443b548bb12f12e119d4b72f314015e67b9e1be97fca", size = 388662, upload-time = "2026-06-30T07:17:38.235Z" }, - { url = "https://files.pythonhosted.org/packages/d3/f4/921c22a4fd0f1c1ac13a3996ffbf0aa67951e2c8ad0d1d9574938a2932e8/rpds_py-2026.6.3-pp311-pypy311_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:127e08c0642d880cf32ca47ec2a4a77b901f7e2dd1ad9762adb13955d72ffcc9", size = 504896, upload-time = "2026-06-30T07:17:39.689Z" }, - { url = "https://files.pythonhosted.org/packages/0b/1b/a114b972cefa1ab1cdb3c7bb177cd3844a12826c507c722d3a73516dbbaf/rpds_py-2026.6.3-pp311-pypy311_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:8bb68f03f395eb793220b45c097bd4d8c32944393da0fad8b999efac0868fc8c", size = 391545, upload-time = "2026-06-30T07:17:41.336Z" }, - { url = "https://files.pythonhosted.org/packages/4e/98/af9b3db77d47fcbe6c8c1f36e2c2147ec70292819e99c325f871584a1c11/rpds_py-2026.6.3-pp311-pypy311_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a3450b693fde92133e9f51060568a4c31fcca76d5e53bbd611e689ca446517e9", size = 380059, upload-time = "2026-06-30T07:17:42.857Z" }, - { url = "https://files.pythonhosted.org/packages/c9/ba/0efd8668b97c1d26a61566386c636a7a7a09829e474fdf807caa15a2c844/rpds_py-2026.6.3-pp311-pypy311_pp73-manylinux_2_31_riscv64.whl", hash = "sha256:5e8d07bddee435a2ff6f1920e18feff28d0bc4533e42f4bf6927fbd073312c41", size = 393235, upload-time = "2026-06-30T07:17:44.637Z" }, - { url = "https://files.pythonhosted.org/packages/62/90/8c139ee9690f73b0829f32647de6f40d826f8f443af6fa72644f96351aac/rpds_py-2026.6.3-pp311-pypy311_pp73-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:3a83ae6c67b7676b9878378547ca8e93ed77a580037bcbcd1d32f739e1e6089c", size = 413008, upload-time = "2026-06-30T07:17:46.225Z" }, - { url = "https://files.pythonhosted.org/packages/9c/97/0043896fdd7828ce09a1d9a8b06433714d0960fc4ff3fc4aa72b666b764e/rpds_py-2026.6.3-pp311-pypy311_pp73-musllinux_1_2_aarch64.whl", hash = "sha256:2bfd04c19ddbd6640de0b51894d764bd2758854d5b75bd102d2ef10cb9c293a9", size = 558118, upload-time = "2026-06-30T07:17:47.759Z" }, - { url = "https://files.pythonhosted.org/packages/f6/40/02355f0e134f783a8f9814c4680a1bd311d37671577a5964ea838573ff37/rpds_py-2026.6.3-pp311-pypy311_pp73-musllinux_1_2_i686.whl", hash = "sha256:ca6546b66be9dc4738b1b043d5ebd5488c66c578c5ff0fd0e8065313fe3afb76", size = 623138, upload-time = "2026-06-30T07:17:49.355Z" }, - { url = "https://files.pythonhosted.org/packages/10/85/48f0abdcef5cce4e034c7a5b0ceeceba0b01bf0d942824f4bb720afe2dec/rpds_py-2026.6.3-pp311-pypy311_pp73-musllinux_1_2_x86_64.whl", hash = "sha256:8e65860d238379ed982fd9ba690579b5e95af2f4840f99c772816dbe573cb826", size = 586486, upload-time = "2026-06-30T07:17:51.141Z" }, -] - [[package]] name = "ruamel-yaml" version = "0.19.1"