Repository navigation
Make main the released 0.7.0 - #16
Merged
Merged
Conversation
Replicated and Shared MergeTree drop an INSERT block identical to an earlier one. After `down` re-inserted a previous head, or `up` re-inserted a revision applied before a downgrade, the row was silently deduplicated while the DELETE of the old head still ran, leaving alembic_version empty so the next `up` replays everything. Every version-row INSERT, from `up`, `down` or `stamp`, now carries its own deduplication token; the official INSERT then waited DELETE transition is otherwise unchanged. Ported from c9d19eb, whose fix lives in the later waiting code. Solves: DRY-1447 Tests: round-trip down/up on Docker replicas (failed before) and single node; version-table integration suite; 573 unit
ClickHouse Cloud routes each TCP connection, not each session id, to a replica, so a session exists only on the replica its connection reached. The migration engine shared clickhouse-connect's default pool with every other in-process client; the session now gets a single-connection pool of its own. A dropped connection still fails closed through session_check. Ported from e320f57; the session body moves into _run_connection to keep _run_online short. Solves: Session routing on ClickHouse Cloud (from DRY-1445) Tests: 573 unit; Docker integration (not cluster) 28 passed
Use the old import solely in the 0.4.1 fixture, keep status a nonblocking reporter, and assert compact waiver output and the SQL statement line rather than the preceding waiver comment. Ported from 602b10e; at the cut the 0.4.1 fixture wrote a ch_migrate import that 0.4.1 cannot load, so both upgrade-env tests errored in setup. Solves: Integration assertions diverged across rename and output cutover Tests: Docker integration (not cluster) 28 passed
Cloud refuses SYSTEM STOP MERGES to every user, keeps Memory rows on the writing replica, and creates Shared* engines. The harness now knows when it runs on Cloud: tests read Memory tables across replicas, expect Shared engines, and skip a held mutation with the Cloud restriction as the reason. Adds the `cloud` marker with checks that only run on Cloud: a SET reaches every later statement whichever replica serves it, and the version table round-trips on SharedMergeTree. Adds scripts/cloud_sweep.py and a README section on running against Cloud. Ported from fad701e without its 0.7 and 0.8 test changes. Solves: DRY-1448, DRY-1413 (0.6 scope) Tests: Docker integration (not cluster) 28 passed, 2 skipped (cloud)
Click could not map the ch_migrate package to the ch-migrate-cli distribution in an editable install (`pip install -e`, `uv sync`) and raised RuntimeError. Installs from PyPI were not affected. Ported from f246ac2. Solves: --version crash in editable installs Tests: --version regression test fails before, passes after
main carries the 0.9 line, so 0.6.x is released from its own branch. GitHub runs release.yml from the tagged commit, so this branch's copy checks that the tag is on dy/ch-migrate-0.6 instead of main; main's copy and its guard are unchanged. The tag must still match __version__. CI also runs on pushes to the branch, so a commit is proven before it is tagged. Solves: Releasing 0.6.0 while main is on the 0.9 line Tests: release.yml parses; job references resolve
up's gate printed findings as `path:2: ERROR [rule] ...` and its errors as click's `Error: ...`, and upgrade-env printed a bare `Warning:` line, while lint and every other command use the ✓/!/✗ markers. The gate now prints lint's own finding line, and commands raise ui.Failure, which click shows as `✗ message` plus hints. Port of main's 7833aaf without its --json test; 0.6 has no JSON output. Solves: CLI output style gap in the 0.6 gate and upgrade-env Tests: tests/test_cli_output.py; gate and session integration assertions updated; unit suite on Python 3.10-3.14
The gate said "Use IF NOT EXISTS" or "Not idempotent by syntax" and left the fix to the reader. Each finding now quotes the statement's own names: the exact IF NOT EXISTS / IF EXISTS rewrite, or, where a waiver is needed, what a retry would do and the usual way to make it safe (TRUNCATE before a backfill, REPLACE PARTITION instead of ATTACH PARTITION ... FROM, fixed values in an UPDATE). Two rules change with it, both checked on ClickHouse 26.3: - RENAME TABLE/DICTIONARY/DATABASE IF EXISTS passes; a retry skips it. - ALTER ... DETACH/ATTACH PARTITION passes. The gate asked for an IF EXISTS form that ClickHouse rejects, though both already converge. A single PART fails on a retry, so it needs a waiver. On Cloud, a retry right after a DROP or RENAME COLUMN fails (code 36) until that column's mutation finishes, even with IF EXISTS. The README recovery note now says to wait for it. Solves: DRY-1246 Tests: corpus + advice unit tests; Docker and Cloud test retries each rewrite after a failure at every statement
0.6.0 ships the session-safe official dialect, the idempotency gate, the SET lint and the version table that cannot re-run a migration, cut from the 1.0 line at dfa17a9 with its later fixes for the same code. A plain pip upgrade reaches 0.6 from 0.5.x, so the CHANGELOG section opens with the upgrade steps and the README says to pin 0.5.1 until each project has run upgrade-env. The README no longer says `status` refuses an old environment: it still reports, which CI reporters rely on. Solves: 0.6.0 release preparation (no tag) Tests: unit, Docker integration and cluster, clean-wheel quick start
The CHANGELOG said the --version crash hit only editable installs and that PyPI installs were not affected. Installing ch-migrate-cli==0.5.1 from PyPI on Python 3.9 and 3.10 shows the same RuntimeError; 0.6.0 from PyPI prints its version. Solves: Inaccurate 0.6.0 CHANGELOG entry (DRY-1246) Tests: Clean PyPI installs of 0.5.1 (3.9, 3.10) and 0.6.0 (3.10)
Preserve accepted-but-unacknowledged mutation evidence and the retention counterexample. Keep the unconditional recovery contract blocked for a decision rather than implementing silent replay. Solves: DRY-1398 evidence and recovery decision record Tests: 16 local probes; 3 crash repeats; 565 tests on Python 3.10-3.14 (cherry picked from commit 0ab51016552e68c2e6e2ee90a3eb2983433a27d0) (cherry picked from commit 924ffd5)
Record Dan’s approved recovery exception without treating missing ownership evidence as completion or permission to replay a write. Keep normal reattachment and the local-only execution boundaries. Solves: DRY-1398 Tests: 16 local probes; 3 crash repeats; 641 tests on Python 3.10-3.14 (cherry picked from commit 22b4c1a57c1a9c41e02e8c0b65eb6e0fa263ef27) (cherry picked from commit f1f42b5)
Waiting decides which statements leave a background mutation by classifying them, so the classifier ships in 0.7 as an internal module. The plan command, its README section and CHANGELOG entry stay on main under DRY-1542. This is the original classifier, without the rebuild case main added later in 780dd4c. (cherry picked from commit 7bd2fd3a, classifier and its tests only) Solves: DRY-1600 Tests: tests/test_classify.py, tests/integration/test_classification.py
Persist intent before submission and resume only proven server work. Poll every required replica in the checked session, preserve completed writes, and fail closed when ownership evidence cannot prove an outcome. Keep the CLI as the owning process and journal version bookkeeping. Solves: DRY-1399 Tests: 655 tests on Python 3.10-3.14; 127 integration; 3 crash repeats (cherry picked from commit 1d52b89677e0d72517a78e255896f7732012e5fb) (cherry picked from commit 39092ef)
Replicated and Shared MergeTree drop an INSERT block identical to an earlier one. After `down` re-inserted a previous head, or `up` re-inserted a revision applied before a downgrade, the row was silently deduplicated while the DELETE of the old head still ran, leaving alembic_version empty so the next `up` replays everything. Each version-row INSERT now carries its own deduplication token, and downgrade version writes take the same path. Cloud ignored insert_deduplicate = 0 in probes; only a unique token kept the row. Solves: DRY-1447 Tests: round-trip down/up on Docker replicas (failed before) and single node; irreversible + version-table suite 5x on Cloud; 715 unit (cherry picked from commit c9d19eb)
Own queue entries with durable markers and require successful results from every target host. Resume the existing entry after interruption, then wait for any mutations it created before completing the revision. Apply the same deadline to cold version-table creation on a cluster. Solves: DRY-1400 Tests: DDL host/crash/TTL cases; 655 tests on Python 3.10-3.14; 137 integration (cherry picked from commit 34d95ebe602ee853024a7a7870247329e38cff68) (cherry picked from commit 5285278)
Keep multi-line waiting and unknown-outcome instructions when SQL execution wraps a server error, while retaining concise database errors. Solves: Operator recovery detail lost in compact upgrade output Tests: Docker mutation failure and unknown-outcome integration cases (cherry picked from commit 9f3b047)
Cloud cannot hold a mutation with SYSTEM STOP MERGES, so the
self-hosted SIGKILL tests skip there. A deliberately slow UPDATE
(sleepEachRow) gives a real window instead. `up` is killed while the
mutation is running, and the rerun must resume from its journal ("not
reissuing SQL"), produce exactly one increment and one owned mutation,
and then advance the version.
Solves: DRY-1401 (W03/W23 on Cloud)
Tests: integration and cloud on the DRY Cloud service, 4 passed
(cherry picked from commit c687095)
main's lint pass (f7b0f6f) turned two checkpoint lambdas into nested functions; taking it keeps the release line's waiting code identical to main's. Solves: DRY-1600 Tests:
The Cloud crash-resume test filters system.mutations with main's REWRITES_PARTS predicate, because SharedMergeTree lists metadata-only ALTERs there too (DRY-1446). Solves: DRY-1600 Tests: tests/integration/test_cloud.py
0.7 ships from its own line, cut from 0.6, while main carries the held 0.8-1.0 work. As on 0.6 (e0e2c3f), this branch's release.yml checks that a tag is on dy/ch-migrate-0.7, and CI runs on pushes to it. main's copy and its guard are unchanged. Solves: DRY-1600 Tests: both workflow files parse
Waiting printed a raw rewritten line (parts_to_do=N) with no sense of how much work was left. Waiters now hand the server's state to one display module: a live bar per mutation in a terminal, weighted by bytes from system.parts and labelled with parts and time left, one row per replica, and a host count for ON CLUSTER DDL. Piped output gets plain lines at the start, each 10% or 30 s, and the finish. The version table's own delete is waited for but not shown. The display never decides completion. Without SELECT on system.parts it counts parts only and says so once; bootstrap now grants it. Failed mutations raise MutationFailed with the reason and KILL statement as fields, and WaitTimeout carries what was pending. Solves: DRY-1601 Tests: tests/test_waiting.py; manual terminal, piped and --timeout runs
Running Alembic in-process (39092ef) dropped 0.6's translated output: up printed Alembic's INFO lines and wrapped every failure in one plain error. The runner now routes Alembic's log records to the same translation runner.py uses for down, labels the waiting display with the migration it belongs to, and reports a failed mutation, a timeout or an unknown outcome as an error with what to do next. Solves: DRY-1601 Tests: tests/test_waiting_display.py; piped and terminal integration tests
0.7.0 is 0.6.0 plus waiting on mutations and ON CLUSTER DDL, with a progress display. The CHANGELOG leads with what changes for a 0.6 project: up now blocks with no default timeout, and roles bootstrapped by 0.6 on Cloud or a cluster need five more grants. No upgrade-env is needed. The README's Cloud retry note now says up waits for the column mutation itself. Solves: DRY-1602 Tests: unit suite; integration suites at 013b33a
"Fix the cause, then run up" only holds for a temporary cause. When the statement itself is wrong (a value that can't be converted, say), any fix is a mutation that queues behind the failed one, and killing it leaves an unknown outcome: re-running or editing the revision refuses until the journal entry is reconciled by hand. The hint now says both, the reason line drops the server version, and the CHANGELOG states the limitation once. Solves: DRY-1601, DRY-1604 (documents current behaviour) Tests: tests/test_waiting_display.py; reproduction on a local 26.3 server
KeyboardInterrupt skipped the runner, so a user who stopped a long wait got click's bare "Aborted!" and a terminal with the bar's cursor still hidden. up now clears the display and says the same as a timeout: the work keeps running, nothing is recorded, and the next up reattaches. Exit status 130. Solves: DRY-1601 Tests: tests/test_waiting_display.py; SIGINT mid-wait then rerun on a local server
The first Cloud run on this branch failed 30 tests: the classifier and waiting tests came over from their original commits, before fad701e taught them about Cloud. SharedMergeTree lists metadata-only ALTERs in system.mutations (DRY-1446), Memory rows and query logs live on one replica, and Cloud refuses SYSTEM STOP MERGES. These are main's copies of conftest.py and both test files, with this branch's progress-output assertions and piped-progress test on top. (cherry picked from commit fad701e5, test files only) Solves: DRY-1602 Tests: Docker integration for both files
Self-hosted migration roles already read every system table through bootstrap's CURRENT GRANTS(SELECT ON system.*), whether single server or cluster. The individual grants matter where that wildcard can't be granted, which is ClickHouse Cloud. Solves: DRY-1602 Tests:
On a shared server such as ClickHouse Cloud, earlier tests' ON CLUSTER commands also read system.distributed_ddl_queue, so the single-node check failed on Cloud without ch-migrate reading the queue at all. It now counts reads after the test starts, on every replica. (cherry picked from commit 41b69ce4, this test only) Solves: DRY-1602 Tests: test_ddl_queue.py on Docker and Cloud
_bar_row took a label, two numbers and the stats text; the numbers now arrive as the ProgressBar that _bar builds, so each helper does one thing. _Wait.observe dropped a timestamp it never read. run_upgrade both configured and ran the upgrade; the config is now _alembic_config. The Progress alias names its types as strings so it can sit with the module constants. No behaviour change. Solves: DRY-1601 (python-style pass) Tests: unit suite
Pure reordering, no other change, so blame on the moved lines stays readable. WaitDisplay now leads waiting_display.py ahead of the progress records it draws, with its private helpers below its public methods; private functions follow the order their callers use them. MutationFailed sits above the FailedMutation it carries, _route_logs above the _run_* helpers that run() calls after it, and the display tests' private helpers below the tests. Solves: DRY-1601 (python-style pass) Tests: unit suite; sorted-line comparison shows only moved lines
With --timeout 1, a slow runner could run out of time while the ON CLUSTER queue entry was still finishing on the second host, so the timeout named the DDL wait instead of the mutation that STOP MERGES holds open, and the parts_to_do assertion failed (CI run 40543073241). Five seconds lets the queue finish first, so the test checks what it means to. main has the same test and the same race. Solves: DRY-1602 Tests: test_ddl_queue.py both_must_finish on the local cluster harness
Eight waiting_*.py modules sat in the package root beside unrelated commands; they are one feature, so they become a subpackage (Dan, 2026-10-27). waiting.py is now waiting/waiter.py, waiting_store.py is waiting/journal.py after the class it holds, and waiting_types.py is split into errors.py and budget.py so no module shadows the standard library's types. The package exports MigrationWaiter, so `from ch_migrate.waiting import MigrationWaiter` still works. Only imports change; no code moves between functions. main still has the flat modules, so the forward-port to main makes the same move. Solves: DRY-1601 (Dan's review) Tests: unit suite; imports resolve from the CLI and env.py
0.7 waited through a journal that recorded which work each statement started, so a rerun never sent a statement twice. That was about 1,300 of the feature's 2,100 lines, duplicated what the idempotency gate already guarantees, and left a killed or rejected mutation unrecoverable without editing the journal by hand (DRY-1604). Dan chose the simpler design on 2026-10-27. After each statement that can write, up now polls system.mutations on the migration session until the project database has no unfinished mutation on any replica that has to apply it. The ids the connected server reports are expected on every replica of a ReplicatedMergeTree table, so a replica that hasn't received one yet is pending, not finished. ON CLUSTER statements wait inside ClickHouse in throw mode, bounded by 270 seconds and what's left of --timeout; wait_end_of_query makes a timeout arrive as an error rather than a truncated response. Work already queued is waited for before the first migration runs, and a rerun waits for unfinished work and then runs the migration again. Removed: the journal and version-write modules, the classifier and its corpus, statement markers, the TTL split and the DDL-queue poller. version_table.py and its tests are 0.6's again. bootstrap grants only system.replicas and system.parts. The README, CHANGELOG and capability 5 describe the new behaviour; the waiting design note is marked superseded. Solves: DRY-1600, DRY-1604 Tests: unit 672; Docker single 53 passed 4 skipped; cluster 16, incl. killed-then-fixed and rejected-then-fixed reruns (DRY-1604)
The status test's docstring named the downstream project whose CI relies on status exiting 0. The contract stands on its own, so it now says what any required CI status job can rely on. Docstring only. Solves: client references in the public repo Tests: unit suite on 3.10-3.14
env.py is about to enforce the same gate for a direct `alembic upgrade`, so the function that prints the pending upgrades' findings and refuses the run moves from the CLI into gate.py, beside lint_pending_up. Moved as is; `ch-migrate up` calls it from there. Solves: DRY-1609 Tests: unit suite in a clean checkout of this commit
The gate read SQL only from string literals passed to op.execute and from run_sql files. op.execute(text(...)), op.get_bind().execute(...) and exec_driver_sql(...) went unchecked, as did any SQL built at run time, so a non-idempotent UPDATE written that way passed with no finding (2 of 5 forms in a probe). Now text() is unwrapped, literal SQL sent through execute or exec_driver_sql on any connection is checked like op.execute, and SQL that only exists at run time becomes an unreadable statement that needs a reasoned waiver. Only the idempotency rule sees unreadable statements; the SQL-parsing rules skip them. Solves: DRY-1609 Tests: unit 675; Docker single node 53 passed, 4 skipped
A rerun runs an interrupted migration again from its first statement. That is safe only for statements the idempotency gate vouches for; a waived UPDATE n = n + 1 whose wait was cut short applied twice and up reported success. Each step is now noted in _ch_migrate_in_progress before it runs and cleared once its version is recorded. A noted step reruns on its own only if every statement passes the gate without a waiver; otherwise up or down stops before sending anything, names the statements, and needs --rerun-interrupted REV (-x rerun_interrupted with alembic). Downgrades are judged by the same rules. The marker table follows the version table's engine (Shared, Replicated database, ON CLUSTER with a Keeper path, or MergeTree), is read with a replica sync or sequential consistency, and is left out of mutation waiting. The same step hook runs the gate before a direct alembic upgrade's first migration; ch-migrate up has already run it. Solves: DRY-1609 Tests: unit 686; Docker single 58 passed, 4 skipped; cluster 16
The timeout and Ctrl-C hints promised that the next up runs the migration again. Since 7c070fa it does that only when the gate vouches for every statement; a waived backfill stops first and asks to be named. The hints now say so. Solves: DRY-1609 Tests: unit 686; piped and terminal progress integration tests
On Cloud the test's own client can reach a replica that hasn't received the failed mutation's reason yet, so reading it once right after up exits found nothing (1 failure in the 7c070fa Cloud run). A probe showed the reason on every replica within 8 seconds; the test now polls for it. up itself is unaffected: it keeps waiting until its replica reports the failure. Solves: DRY-1609 Tests: the test on Docker, and three passing runs on Cloud
Reordering only; every line is unchanged. RerunGuard is the reason reruns.py exists, so it now leads the module, followed by the value types in the order it uses them, and _acknowledged follows _started, as begin calls them. In env.py, _rerun_guard and _enforce_gate move below _checked_session and _post_migrate, which _run_connection calls first. In gate.py, lint_pending_up moves above print_gate_findings, matching enforce_up_gate's calls. Solves: python-style pass over DRY-1609 Tests: unit 686 on 3.10 and 3.14; Docker single node 58 passed, 4 skipped; cluster 16 passed
The test reached 36 lines once it checked the stop and the named rerun. Starting up and killing it once its mutation runs is one step, so it moves to _kill_up_once_its_mutation_runs and the test reads as the scenario. Solves: python-style pass over DRY-1609 Tests: Cloud "integration and cloud" 4 passed (CLOUD_SIGKILL value=1, mutations=2); no leftover databases
Formatting only. pyproject.toml configures black at 100 columns, but 14 files in ch_migrate/ and tests/ weren't formatted, so each one touched in a later change dragged a reformat into that diff (as lint.py did in c07fcbc). Now `black --check ch_migrate tests` passes. The archived spike scripts under docs/design/spikes/ and the 0.4.1 env.py fixture stay as they were written. Solves: python-style pass over DRY-1609 Tests: unit 686 on 3.10-3.14; Docker single node 58 passed, 4 skipped; cluster 16 passed
main now holds the code released as 0.7.0 from dy/ch-migrate-0.7 (tag v0.7.0). The 1.0 work that was on main (plan, --json, rebuild_table, the drift gate and the journal waiter) stays intact on dy/ch-migrate-1.0 at efca090 while DRY-1542 decides each feature. This is a merge that keeps 0.7's tree, so main moves forward without a force push. main keeps its own ci.yml and release.yml, so release tags are still checked against main. Solves: DRY-1601, DRY-1602 (main matches the release line) Tests: tree equals 3367b9d except .github/workflows; CI on the PR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes
mainthe released 0.7.0. CI runs here; the change lands as a fast-forward push of this merge commit, not through the merge button. The repo's UI allows only squash or rebase merges, and either would replay the release line's commits.f740ec7has parents3367b9d(tagv0.7.0) andefca090(currentmain). Its tree is 0.7.0 exactly, except.github/workflows, which staysmain's, so release tags are still checked againstmain.main(plan,--json,rebuild_table, the drift gate and the journal waiter) is kept ondy/ch-migrate-1.0, fast-forwarded toefca090. DRY-1542 decides what happens to each feature.main's CHANGELOG now has the 0.6.0 and 0.7.0 sections, and__version__is0.7.0. The untagged 0.9.0 notes stay on the 1.0 branch.🤖 Generated with Claude Code