Repository navigation
Conversation
Preserve accepted-but-unacknowledged mutation evidence and the retention counterexample. Keep the unconditional recovery contract blocked for a decision rather than implementing silent replay. Solves: DRY-1398 evidence and recovery decision record Tests: 16 local probes; 3 crash repeats; 565 tests on Python 3.10-3.14 (cherry picked from commit 0ab51016552e68c2e6e2ee90a3eb2983433a27d0) (cherry picked from commit 924ffd5)
Record Dan’s approved recovery exception without treating missing ownership evidence as completion or permission to replay a write. Keep normal reattachment and the local-only execution boundaries. Solves: DRY-1398 Tests: 16 local probes; 3 crash repeats; 641 tests on Python 3.10-3.14 (cherry picked from commit 22b4c1a57c1a9c41e02e8c0b65eb6e0fa263ef27) (cherry picked from commit f1f42b5)
Waiting decides which statements leave a background mutation by classifying them, so the classifier ships in 0.7 as an internal module. The plan command, its README section and CHANGELOG entry stay on main under DRY-1542. This is the original classifier, without the rebuild case main added later in 780dd4c. (cherry picked from commit 7bd2fd3a, classifier and its tests only) Solves: DRY-1600 Tests: tests/test_classify.py, tests/integration/test_classification.py
Persist intent before submission and resume only proven server work. Poll every required replica in the checked session, preserve completed writes, and fail closed when ownership evidence cannot prove an outcome. Keep the CLI as the owning process and journal version bookkeeping. Solves: DRY-1399 Tests: 655 tests on Python 3.10-3.14; 127 integration; 3 crash repeats (cherry picked from commit 1d52b89677e0d72517a78e255896f7732012e5fb) (cherry picked from commit 39092ef)
Replicated and Shared MergeTree drop an INSERT block identical to an earlier one. After `down` re-inserted a previous head, or `up` re-inserted a revision applied before a downgrade, the row was silently deduplicated while the DELETE of the old head still ran, leaving alembic_version empty so the next `up` replays everything. Each version-row INSERT now carries its own deduplication token, and downgrade version writes take the same path. Cloud ignored insert_deduplicate = 0 in probes; only a unique token kept the row. Solves: DRY-1447 Tests: round-trip down/up on Docker replicas (failed before) and single node; irreversible + version-table suite 5x on Cloud; 715 unit (cherry picked from commit c9d19eb)
Own queue entries with durable markers and require successful results from every target host. Resume the existing entry after interruption, then wait for any mutations it created before completing the revision. Apply the same deadline to cold version-table creation on a cluster. Solves: DRY-1400 Tests: DDL host/crash/TTL cases; 655 tests on Python 3.10-3.14; 137 integration (cherry picked from commit 34d95ebe602ee853024a7a7870247329e38cff68) (cherry picked from commit 5285278)
Keep multi-line waiting and unknown-outcome instructions when SQL execution wraps a server error, while retaining concise database errors. Solves: Operator recovery detail lost in compact upgrade output Tests: Docker mutation failure and unknown-outcome integration cases (cherry picked from commit 9f3b047)
Cloud cannot hold a mutation with SYSTEM STOP MERGES, so the
self-hosted SIGKILL tests skip there. A deliberately slow UPDATE
(sleepEachRow) gives a real window instead. `up` is killed while the
mutation is running, and the rerun must resume from its journal ("not
reissuing SQL"), produce exactly one increment and one owned mutation,
and then advance the version.
Solves: DRY-1401 (W03/W23 on Cloud)
Tests: integration and cloud on the DRY Cloud service, 4 passed
(cherry picked from commit c687095)
main's lint pass (f7b0f6f) turned two checkpoint lambdas into nested functions; taking it keeps the release line's waiting code identical to main's. Solves: DRY-1600 Tests:
The Cloud crash-resume test filters system.mutations with main's REWRITES_PARTS predicate, because SharedMergeTree lists metadata-only ALTERs there too (DRY-1446). Solves: DRY-1600 Tests: tests/integration/test_cloud.py
0.7 ships from its own line, cut from 0.6, while main carries the held 0.8-1.0 work. As on 0.6 (e0e2c3f), this branch's release.yml checks that a tag is on dy/ch-migrate-0.7, and CI runs on pushes to it. main's copy and its guard are unchanged. Solves: DRY-1600 Tests: both workflow files parse
Waiting printed a raw rewritten line (parts_to_do=N) with no sense of how much work was left. Waiters now hand the server's state to one display module: a live bar per mutation in a terminal, weighted by bytes from system.parts and labelled with parts and time left, one row per replica, and a host count for ON CLUSTER DDL. Piped output gets plain lines at the start, each 10% or 30 s, and the finish. The version table's own delete is waited for but not shown. The display never decides completion. Without SELECT on system.parts it counts parts only and says so once; bootstrap now grants it. Failed mutations raise MutationFailed with the reason and KILL statement as fields, and WaitTimeout carries what was pending. Solves: DRY-1601 Tests: tests/test_waiting.py; manual terminal, piped and --timeout runs
Running Alembic in-process (39092ef) dropped 0.6's translated output: up printed Alembic's INFO lines and wrapped every failure in one plain error. The runner now routes Alembic's log records to the same translation runner.py uses for down, labels the waiting display with the migration it belongs to, and reports a failed mutation, a timeout or an unknown outcome as an error with what to do next. Solves: DRY-1601 Tests: tests/test_waiting_display.py; piped and terminal integration tests
0.7.0 is 0.6.0 plus waiting on mutations and ON CLUSTER DDL, with a progress display. The CHANGELOG leads with what changes for a 0.6 project: up now blocks with no default timeout, and roles bootstrapped by 0.6 on Cloud or a cluster need five more grants. No upgrade-env is needed. The README's Cloud retry note now says up waits for the column mutation itself. Solves: DRY-1602 Tests: unit suite; integration suites at 013b33a
"Fix the cause, then run up" only holds for a temporary cause. When the statement itself is wrong (a value that can't be converted, say), any fix is a mutation that queues behind the failed one, and killing it leaves an unknown outcome: re-running or editing the revision refuses until the journal entry is reconciled by hand. The hint now says both, the reason line drops the server version, and the CHANGELOG states the limitation once. Solves: DRY-1601, DRY-1604 (documents current behaviour) Tests: tests/test_waiting_display.py; reproduction on a local 26.3 server
KeyboardInterrupt skipped the runner, so a user who stopped a long wait got click's bare "Aborted!" and a terminal with the bar's cursor still hidden. up now clears the display and says the same as a timeout: the work keeps running, nothing is recorded, and the next up reattaches. Exit status 130. Solves: DRY-1601 Tests: tests/test_waiting_display.py; SIGINT mid-wait then rerun on a local server
The first Cloud run on this branch failed 30 tests: the classifier and waiting tests came over from their original commits, before fad701e taught them about Cloud. SharedMergeTree lists metadata-only ALTERs in system.mutations (DRY-1446), Memory rows and query logs live on one replica, and Cloud refuses SYSTEM STOP MERGES. These are main's copies of conftest.py and both test files, with this branch's progress-output assertions and piped-progress test on top. (cherry picked from commit fad701e5, test files only) Solves: DRY-1602 Tests: Docker integration for both files
Self-hosted migration roles already read every system table through bootstrap's CURRENT GRANTS(SELECT ON system.*), whether single server or cluster. The individual grants matter where that wildcard can't be granted, which is ClickHouse Cloud. Solves: DRY-1602 Tests:
On a shared server such as ClickHouse Cloud, earlier tests' ON CLUSTER commands also read system.distributed_ddl_queue, so the single-node check failed on Cloud without ch-migrate reading the queue at all. It now counts reads after the test starts, on every replica. (cherry picked from commit 41b69ce4, this test only) Solves: DRY-1602 Tests: test_ddl_queue.py on Docker and Cloud
_bar_row took a label, two numbers and the stats text; the numbers now arrive as the ProgressBar that _bar builds, so each helper does one thing. _Wait.observe dropped a timestamp it never read. run_upgrade both configured and ran the upgrade; the config is now _alembic_config. The Progress alias names its types as strings so it can sit with the module constants. No behaviour change. Solves: DRY-1601 (python-style pass) Tests: unit suite
Pure reordering, no other change, so blame on the moved lines stays readable. WaitDisplay now leads waiting_display.py ahead of the progress records it draws, with its private helpers below its public methods; private functions follow the order their callers use them. MutationFailed sits above the FailedMutation it carries, _route_logs above the _run_* helpers that run() calls after it, and the display tests' private helpers below the tests. Solves: DRY-1601 (python-style pass) Tests: unit suite; sorted-line comparison shows only moved lines
With --timeout 1, a slow runner could run out of time while the ON CLUSTER queue entry was still finishing on the second host, so the timeout named the DDL wait instead of the mutation that STOP MERGES holds open, and the parts_to_do assertion failed (CI run 40543073241). Five seconds lets the queue finish first, so the test checks what it means to. main has the same test and the same race. Solves: DRY-1602 Tests: test_ddl_queue.py both_must_finish on the local cluster harness
Eight waiting_*.py modules sat in the package root beside unrelated commands; they are one feature, so they become a subpackage (Dan, 2026-10-27). waiting.py is now waiting/waiter.py, waiting_store.py is waiting/journal.py after the class it holds, and waiting_types.py is split into errors.py and budget.py so no module shadows the standard library's types. The package exports MigrationWaiter, so `from ch_migrate.waiting import MigrationWaiter` still works. Only imports change; no code moves between functions. main still has the flat modules, so the forward-port to main makes the same move. Solves: DRY-1601 (Dan's review) Tests: unit suite; imports resolve from the CLI and env.py
0.7 waited through a journal that recorded which work each statement started, so a rerun never sent a statement twice. That was about 1,300 of the feature's 2,100 lines, duplicated what the idempotency gate already guarantees, and left a killed or rejected mutation unrecoverable without editing the journal by hand (DRY-1604). Dan chose the simpler design on 2026-10-27. After each statement that can write, up now polls system.mutations on the migration session until the project database has no unfinished mutation on any replica that has to apply it. The ids the connected server reports are expected on every replica of a ReplicatedMergeTree table, so a replica that hasn't received one yet is pending, not finished. ON CLUSTER statements wait inside ClickHouse in throw mode, bounded by 270 seconds and what's left of --timeout; wait_end_of_query makes a timeout arrive as an error rather than a truncated response. Work already queued is waited for before the first migration runs, and a rerun waits for unfinished work and then runs the migration again. Removed: the journal and version-write modules, the classifier and its corpus, statement markers, the TTL split and the DDL-queue poller. version_table.py and its tests are 0.6's again. bootstrap grants only system.replicas and system.parts. The README, CHANGELOG and capability 5 describe the new behaviour; the waiting design note is marked superseded. Solves: DRY-1600, DRY-1604 Tests: unit 672; Docker single 53 passed 4 skipped; cluster 16, incl. killed-then-fixed and rejected-then-fixed reruns (DRY-1604)
The status test's docstring named the downstream project whose CI relies on status exiting 0. The contract stands on its own, so it now says what any required CI status job can rely on. Docstring only. Solves: client references in the public repo Tests: unit suite on 3.10-3.14
env.py is about to enforce the same gate for a direct `alembic upgrade`, so the function that prints the pending upgrades' findings and refuses the run moves from the CLI into gate.py, beside lint_pending_up. Moved as is; `ch-migrate up` calls it from there. Solves: DRY-1609 Tests: unit suite in a clean checkout of this commit
The gate read SQL only from string literals passed to op.execute and from run_sql files. op.execute(text(...)), op.get_bind().execute(...) and exec_driver_sql(...) went unchecked, as did any SQL built at run time, so a non-idempotent UPDATE written that way passed with no finding (2 of 5 forms in a probe). Now text() is unwrapped, literal SQL sent through execute or exec_driver_sql on any connection is checked like op.execute, and SQL that only exists at run time becomes an unreadable statement that needs a reasoned waiver. Only the idempotency rule sees unreadable statements; the SQL-parsing rules skip them. Solves: DRY-1609 Tests: unit 675; Docker single node 53 passed, 4 skipped
A rerun runs an interrupted migration again from its first statement. That is safe only for statements the idempotency gate vouches for; a waived UPDATE n = n + 1 whose wait was cut short applied twice and up reported success. Each step is now noted in _ch_migrate_in_progress before it runs and cleared once its version is recorded. A noted step reruns on its own only if every statement passes the gate without a waiver; otherwise up or down stops before sending anything, names the statements, and needs --rerun-interrupted REV (-x rerun_interrupted with alembic). Downgrades are judged by the same rules. The marker table follows the version table's engine (Shared, Replicated database, ON CLUSTER with a Keeper path, or MergeTree), is read with a replica sync or sequential consistency, and is left out of mutation waiting. The same step hook runs the gate before a direct alembic upgrade's first migration; ch-migrate up has already run it. Solves: DRY-1609 Tests: unit 686; Docker single 58 passed, 4 skipped; cluster 16
The timeout and Ctrl-C hints promised that the next up runs the migration again. Since 7c070fa it does that only when the gate vouches for every statement; a waived backfill stops first and asks to be named. The hints now say so. Solves: DRY-1609 Tests: unit 686; piped and terminal progress integration tests
On Cloud the test's own client can reach a replica that hasn't received the failed mutation's reason yet, so reading it once right after up exits found nothing (1 failure in the 7c070fa Cloud run). A probe showed the reason on every replica within 8 seconds; the test now polls for it. up itself is unaffected: it keeps waiting until its replica reports the failure. Solves: DRY-1609 Tests: the test on Docker, and three passing runs on Cloud
Reordering only; every line is unchanged. RerunGuard is the reason reruns.py exists, so it now leads the module, followed by the value types in the order it uses them, and _acknowledged follows _started, as begin calls them. In env.py, _rerun_guard and _enforce_gate move below _checked_session and _post_migrate, which _run_connection calls first. In gate.py, lint_pending_up moves above print_gate_findings, matching enforce_up_gate's calls. Solves: python-style pass over DRY-1609 Tests: unit 686 on 3.10 and 3.14; Docker single node 58 passed, 4 skipped; cluster 16 passed
The test reached 36 lines once it checked the stop and the named rerun. Starting up and killing it once its mutation runs is one step, so it moves to _kill_up_once_its_mutation_runs and the test reads as the scenario. Solves: python-style pass over DRY-1609 Tests: Cloud "integration and cloud" 4 passed (CLOUD_SIGKILL value=1, mutations=2); no leftover databases
Formatting only. pyproject.toml configures black at 100 columns, but 14 files in ch_migrate/ and tests/ weren't formatted, so each one touched in a later change dragged a reformat into that diff (as lint.py did in c07fcbc). Now `black --check ch_migrate tests` passes. The archived spike scripts under docs/design/spikes/ and the 0.4.1 env.py fixture stay as they were written. Solves: python-style pass over DRY-1609 Tests: unit 686 on 3.10-3.14; Docker single node 58 passed, 4 skipped; cluster 16 passed
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Do not merge. This PR exists only to run CI on the 0.7.0 release line.
dy/ch-migrate-0.7is released by tagging it, never by merging it into the 0.6 line.What the branch holds, on top of 0.6.0:
upwaits before it records a migration: after each statement that can write, it pollssystem.mutationsuntil the project database has no unfinished mutation on any replica that must apply it, andON CLUSTERDDL waits inside ClickHouse in throw mode.--timeoutbounds the wait.__version__ = "0.7.0"and the CHANGELOG 0.7.0 section.release.ymlacceptingv0.7.xtags from this branch.Tracked in DRY-1600, DRY-1601, DRY-1602 and DRY-1604 (internal).
🤖 Generated with Claude Code