DO NOT MERGE: 0.7.0 release line (CI only)

.github/workflows/ci.yml

@@ -5,7 +5,7 @@ name: ci
 
 on:
   push:
-    branches: [main, dy/ch-migrate-0.6]
+    branches: [main, dy/ch-migrate-0.7]
   pull_request:
   workflow_call:
 

.github/workflows/release.yml

@@ -8,13 +8,13 @@ name: release
 # PyPI publisher for project ch-migrate-cli: owner DRYCodeWorks, repository
 # ch-migrate, workflow release.yml, no environment.
 #
-# This copy of the workflow is on the 0.6.x release branch, dy/ch-migrate-0.6.
-# GitHub runs the workflow file from the tagged commit, so 0.6.x tags are
+# This copy of the workflow is on the 0.7.x release branch, dy/ch-migrate-0.7.
+# GitHub runs the workflow file from the tagged commit, so 0.7.x tags are
 # checked against this branch while tags on main keep main's own check.
 #
-# To release 0.6.x: set __version__ in ch_migrate/__init__.py, commit, push
-# dy/ch-migrate-0.6, then `git tag vX.Y.Z && git push origin vX.Y.Z`. Tags on
-# commits that are not on dy/ch-migrate-0.6 are refused before anything builds.
+# To release 0.7.x: set __version__ in ch_migrate/__init__.py, commit, push
+# dy/ch-migrate-0.7, then `git tag vX.Y.Z && git push origin vX.Y.Z`. Tags on
+# commits that are not on dy/ch-migrate-0.7 are refused before anything builds.
 
 on:
   push:
@@ -24,7 +24,7 @@ permissions:
   contents: read
 
 env:
-  RELEASE_BRANCH: dy/ch-migrate-0.6
+  RELEASE_BRANCH: dy/ch-migrate-0.7
 
 jobs:
   on-release-branch:

CHANGELOG.md

@@ -4,6 +4,38 @@ Changes are recorded in [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
 
 ## [Unreleased]
 
+## [0.7.0]
+
+0.7 waits for the work a migration starts. `up` doesn't record a migration as applied until the mutations and `ON CLUSTER` DDL its statements started have finished, and it shows their progress while it waits. A normal `pip install -U` reaches it from 0.6, so read this before upgrading:
+
+1. **`up` now blocks until that work finishes, with no default timeout.** If a CI job's time limit stops it mid-wait, ClickHouse keeps working, and the next `up` waits for that work before it runs the migration again. `--timeout SECONDS` sets a limit of your own.
+2. **A rerun runs the stopped migration again from its first statement, but only on its own when every statement passes the idempotency gate without a waiver.** An interrupted migration with a waived statement, which includes every `UPDATE`, `DELETE` and `INSERT ... SELECT`, stops the next run until you check what the earlier one changed and name it with `--rerun-interrupted REV`. CI jobs that retry `up` after a timeout need that step.
+3. **The gate now reads more of your migrations.** SQL in `text("...")` and SQL sent through `op.get_bind().execute(...)` or `exec_driver_sql(...)` is checked, and SQL built at run time needs a waiver. `lint` and `up` can report migrations that passed before; waive them or keep them behind `lint.gate_baseline`.
+4. **Two grants, only where `bootstrap` couldn't grant `SELECT ON system.*`**, such as on ClickHouse Cloud. As the admin user, replacing `<project>` with your project name:
+   ```sql
+   GRANT CURRENT GRANTS(SELECT ON system.parts) TO <project>_migration_role;
+   GRANT CURRENT GRANTS(SELECT ON system.replicas) TO <project>_migration_role;
+   ```
+   `system.parts` gives the progress bar its bytes and time estimate; without it the bar counts parts. `system.replicas` is read only for self-hosted replicated tables, so Cloud needs just the first.
+5. **No `upgrade-env` this time.** The environment 0.6 wrote is current. `up` creates one small table, `_ch_migrate_in_progress`, in the project database.
+
+To stay on 0.6, pin `ch-migrate-cli==0.6.0`.
+
+### Changed
+
+- `up` waits before it records a migration as applied. After each statement that can write, it polls `system.mutations` on the migration session until the project database has no unfinished mutation on any replica that has to apply it, instead of holding a request open with `mutations_sync`. Mutations another writer started count too, and ones already queued are waited for before the first migration runs. The version table's own mutations stay with its 0.6 bookkeeping.
+- `ON CLUSTER` statements wait inside ClickHouse: `up` sends them with `distributed_ddl_output_mode = 'throw'` and a limit of 270 seconds, or what's left of `--timeout` if that's less, so a host that fails or doesn't finish in time fails the migration with ClickHouse's message.
+- A failed mutation fails the migration with ClickHouse's `latest_fail_reason` and prints the scoped `KILL MUTATION` statement for an operator; `ch-migrate` never kills a mutation itself. A failed mutation from before the run blocks it the same way. After a kill, fix the SQL and run `up` again, naming the revision with `--rerun-interrupted` if it's waived.
+- `up` runs Alembic in its own process, so killing `up` can't leave a child process advancing the version table. `down` and a direct `alembic upgrade` go through the same environment and wait too; `--timeout` exists only on `ch-migrate up`.
+- The idempotency gate checks literal SQL in `text("...")` and in `execute`/`exec_driver_sql` on any connection, requires a waiver for SQL that only exists at run time, and runs from `env.py` for a direct `alembic upgrade`.
+
+### Added
+
+- `up --timeout SECONDS`: one waiting budget for the whole run. When it runs out, `up` exits 1, names what is still pending and leaves the work running; the next `up` waits on it again, then runs the migration.
+- Progress while `up` waits: in a terminal, a live bar per mutation weighted by bytes from `system.parts`, with parts and time left, and one row per replica. Piped output gets plain lines when a wait starts, at each 10% or every 30 seconds, and when it finishes. Without access to `system.parts` the bar counts parts only.
+- `up --rerun-interrupted REV` and `down --rerun-interrupted REV` (`-x rerun_interrupted=REV` with `alembic`): run an interrupted migration again after checking what its earlier run changed. Each migration is noted in `_ch_migrate_in_progress` while it runs, created with the version table's engine.
+- `bootstrap` grants `SELECT` on `system.replicas` and `system.parts`.
+
 ## [0.6.0]
 
 0.6 runs migrations on ClickHouse's official clickhouse-connect dialect in one session-safe connection, and stops a migration from running twice. A normal `pip install -U` reaches it from 0.5.x, so read this before upgrading:

README.md

@@ -23,8 +23,15 @@ This README describes the source checkout, which may be ahead of PyPI. To try an
 
 ## Upgrade from 0.x
 
-0.6 requires Python 3.10+; 0.5.x is the last line supporting Python 3.9.
-A loose requirement such as `ch-migrate-cli` or `~=0.5` upgrades to 0.6; pin
+**From 0.6:** 0.7 needs no `upgrade-env`. `up` now waits for the mutations and
+`ON CLUSTER` DDL its statements start (see [Migrations that wait](#migrations-that-wait)),
+so give CI jobs time to finish or pass `--timeout`. Where `bootstrap` couldn't grant
+`SELECT ON system.*`, such as on ClickHouse Cloud, add the grants listed in the
+[CHANGELOG](CHANGELOG.md). A loose requirement such as `ch-migrate-cli` upgrades to
+0.7; pin `ch-migrate-cli==0.6.0` to stay on 0.6.
+
+**From 0.5:** 0.6 and later require Python 3.10+; 0.5.x is the last line supporting Python 3.9.
+A loose requirement such as `ch-migrate-cli` or `~=0.5` upgrades past 0.5; pin
 `ch-migrate-cli==0.5.1` until each project has run the command below.
 After upgrading the package, run this in each migration project:
 
@@ -227,13 +234,111 @@ information or the single-server/Cloud default for version-table DDL. Review
 that DDL before using it for a self-hosted Replicated database. Cloud qualification
 is separate from the local suite; the local replicated harness has one shard.
 
+### Migrations that wait
+
+`up` doesn't record a migration as applied until the work its statements started
+has finished. After each statement that can write, it polls `system.mutations` on
+the same checked ClickHouse session, so earlier `SET` statements stay in effect.
+`op.execute`, raw `op.get_bind()` execution and `run_sql` all pass through that
+check. Upgrades run in the CLI process, so killing it can't leave a child Alembic
+runner advancing revisions. `down` waits the same way.
+
+```bash
+ch-migrate up dev                 # No default waiting timeout
+ch-migrate up dev --timeout 120   # One waiting budget across the invocation
+```
+
+What `up` waits for:
+
+- **Every unfinished mutation in the project database,** including one another
+  writer started. Before the first migration runs, `up` waits for any already
+  queued, and a failed one blocks the run. The version table's own mutations are
+  left to [its bookkeeping](#the-version-table).
+- **Each replica that has to apply it.** Zero parts left isn't completion on its
+  own: the mutation must report `is_done`. A ReplicatedMergeTree mutation must be
+  finished on every replica, and a replica that hasn't received it yet is still
+  pending. Configure `cluster` for self-hosted replicated tables, or use a
+  Replicated database's automatic cluster; without either, `up` refuses a table
+  with more than one replica. SharedMergeTree on ClickHouse Cloud reports one
+  shared mutation.
+- **`ON CLUSTER` DDL on every host.** `up` sends these statements with
+  `distributed_ddl_output_mode = 'throw'`, so ClickHouse answers once every host
+  has run the statement, and a host that fails or doesn't finish in time fails the
+  migration with ClickHouse's message. The limit is 270 seconds, below the
+  client's read timeout, or what's left of `--timeout` if that's less. Mutations
+  the statement starts are then waited for as above.
+
+Statements ClickHouse already holds until they finish, such as a type-changing
+`MODIFY COLUMN`, a `MODIFY TTL` that materializes, or a lightweight `DELETE`,
+still hold their request as they did in 0.6.
+
+While it waits, `up` shows progress on stderr. In a terminal, each mutation gets a
+live bar weighted by bytes, with its parts and an estimate of the time left:
+
+```text
+→ Applying 3f9c2a1b  backfill_digest
+  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━  18.2 / 41.0 GiB · 23/41 parts · 1:12 · ~1:30 left
+✓ 3f9c2a1b  backfill_digest  mutation_7.txt · 41 parts · 41.0 GiB · 2:03
+```
+
+Self-hosted replicated tables get one row per replica. When output isn't a
+terminal (CI logs, pipes, coding agents), `up` prints plain lines instead: one
+when a wait starts, one at each 10% of the parts or every 30 seconds, and one
+when it finishes:
+
+```text
+→ Applying 3f9c2a1b  backfill_digest
+  waiting on mutation_7.txt (demo_dev.events): 0/41 parts
+  waiting on mutation_7.txt (demo_dev.events): 5/41 parts · 0:14
+✓ 3f9c2a1b  backfill_digest  mutation_7.txt · 41 parts · 2:03
+```
+
+Part sizes come from `system.parts`. If the migration role can't read it, the bar
+counts parts only and says so once; progress never changes whether `up` succeeds.
+
+**When `up` stops short.** After a timeout, Ctrl-C, dropped connection, killed
+process or failed statement, ClickHouse keeps working and the migration isn't
+recorded. `up` and `down` note each migration in `_ch_migrate_in_progress` when it
+starts and clear the note once its version is recorded, so the next run knows the
+migration was interrupted. That run waits for unfinished work, then:
+
+- **Runs the migration again on its own** when every statement in it passes the
+  idempotency gate without a waiver, so a repeat leaves the same result.
+- **Stops before sending anything** when the migration has a waived statement or
+  SQL the gate can't read, because that statement may already have applied. It
+  names each one. Check what the earlier run changed, fix the SQL if needed, then
+  name the revision: `ch-migrate up ENV --rerun-interrupted REV` (or `down`), or
+  `-x rerun_interrupted=REV` with `alembic`. Naming it accepts the repeat.
+
+Every `UPDATE`, `DELETE` and `INSERT ... SELECT` needs a waiver, so an interrupted
+backfill always takes that second path.
+
+**Failed mutation.** ClickHouse's `latest_fail_reason` stops the run, and `up`
+prints the scoped `KILL MUTATION` statement for an operator; ch-migrate never
+runs it. ClickHouse keeps retrying a failed mutation, and later mutations on that
+table queue behind it. If the cause is temporary, such as memory or disk, fix it
+and run `up` again. If the statement itself is wrong, kill the mutation, fix the
+SQL and run `up` again, naming the revision with `--rerun-interrupted` if it's
+waived. Killing a mutation doesn't undo the parts it already rewrote.
+
+Run one migration runner per database, using a CI concurrency key or an external
+lock; the in-progress notes aren't a lock. Re-run `bootstrap` when upgrading a
+restricted migration user to obtain the `system.replicas` and `system.parts` grants.
+
 ### Re-runnable migrations
 
 `up` statically checks pending upgrade statements before Alembic executes any of
 them. Idempotency and standalone-SET errors refuse the whole run and print the source file, line,
 statement, and suggested fix. There is no command-line bypass; `--skip-mv-check`
 does not skip this gate. Other findings are warnings in `up`; standalone `lint`
-retains their configured severities.
+retains their configured severities. A direct `alembic upgrade` runs the same gate
+from `env.py` before its first migration.
+
+The gate reads SQL in `run_sql` files and in literal strings, f-strings and
+`text("...")` passed to `op.execute`, or to `execute` and `exec_driver_sql` on a
+connection such as `op.get_bind()`. SQL that only exists at run time, such as a
+variable, a built string or `run_sql(name)`, can't be checked, so it needs a
+reasoned waiver like any other statement the gate can't vouch for.
 
 Static lint rejects a standalone `SET` when `migrations/env.py` lacks the
 `CH_MIGRATE_ENV_VERSION = 2` marker: that statement would be ignored on the old
@@ -267,7 +372,7 @@ INSERT INTO {db}.target SELECT id FROM {db}.source;
 ```
 
 For inline Python SQL, use the same directive in a `#` comment directly above
-`op.execute`. Empty reasons remain errors. Waivers print as INFO with their
+the call that sends it. Empty reasons remain errors. Waivers print as INFO with their
 reasons in `lint` and `up`; they do not make the statement idempotent.
 `MODIFY`, `MATERIALIZE`, `CLEAR COLUMN`, `TRUNCATE`, `OPTIMIZE`, `SYSTEM`, grants,
 revokes, `REPLACE PARTITION`, attaching or detaching a whole partition, and
@@ -276,11 +381,12 @@ The reviewable classification corpus is `tests/corpus/idempotency.yaml`.
 
 **Recovery:** fix the cause of a partial failure, then run `ch-migrate up ENV`
 again. Earlier idempotent statements can run again; the version advances only
-when the revision completes. On ClickHouse Cloud, a column that the failed attempt
-dropped or renamed stays locked until that change's mutation finishes: a retry
-before then fails with "affected by mutation ... which is not finished yet", even
-with `IF EXISTS`. Wait for it in `system.mutations`, then run `up` again. Before retrying waived operations, review what the
-earlier attempt actually changed.
+when the revision completes. A dropped or renamed column stays locked until that
+change's mutation finishes, and `up` waits for it before running the next
+statement, so a retry after a later failure doesn't hit "affected by mutation ...
+which is not finished yet". If `up` was stopped while it waited, run it again: it
+waits for that mutation, then runs the revision again, unless the revision has a
+statement the gate can't vouch for (see below).
 
 **Existing history:** `upgrade-env` records the current local script head(s) in
 `config.yaml` as `lint.gate_baseline`, preserving YAML comments. It records
@@ -393,7 +499,7 @@ Example: `ch-migrate new dev add_status --table logs`
 
 ### `up`
 
-`ch-migrate up ENV [-r REV] [--skip-mv-check] [--verbose]` applies migrations to `head` by default after the idempotency gate passes, printing one line per migration. `-r/--revision` selects a target. `--skip-mv-check` skips nonblocking materialized-view declaration checks, not the idempotency gate; use it only after reviewing those findings. If a migration fails, `up` names it, the SQL file, the statement and its line, and ClickHouse's error; `--verbose` adds the Python traceback.
+`ch-migrate up ENV [-r REV] [--timeout SECONDS] [--skip-mv-check] [--verbose]` applies migrations to `head` by default after the idempotency gate passes, printing one line per migration. `-r/--revision` selects a target. `--timeout` is a finite positive waiting limit; omitting it waits without a deadline. `--skip-mv-check` skips nonblocking materialized-view declaration checks, not the idempotency gate; use it only after reviewing those findings. If a migration fails, `up` names it, the SQL file, the statement and its line, and ClickHouse's error; `--verbose` adds the Python traceback.
 
 Example: `ch-migrate up dev --revision abc123`
 
@@ -507,7 +613,13 @@ The legacy `CH_<ENV>_PASSWORD` remains supported. Never commit credentials or pa
 
 ### Hooks
 
-Top-level hooks run SQL on the migration connection. `pre_migrate` runs before the migration batch; `post_migrate` runs after each revision. `{db}` is substituted. Hooks execute as SQLAlchemy text, not through the SQL-file splitter; supply one statement per entry. Hook SQL is logged, so do not put secrets in it.
+Top-level hooks run SQL on the migration connection. `pre_migrate` runs before the
+migration batch; `post_migrate` runs after each revision body **before its version
+is completed**, so hook mutations cannot escape the waiting barrier. A resumed
+upgrade reuses its unfinished batch's receipts instead of repeating completed
+hook writes. `{db}` is substituted. Hooks execute as SQLAlchemy text, not through
+the SQL-file splitter; supply one statement per entry. Hook SQL is logged, so do
+not put secrets in it.
 
 ```yaml
 hooks:

ch_migrate/__init__.py

@@ -5,7 +5,7 @@
     from ch_migrate import read_sql, get_db, get_env_config, create_dictionary
 """
 
-__version__ = "0.6.0"
+__version__ = "0.7.0"
 
 
 from typing import Any