Repository navigation
storage/sql-server: Catch dropped and altered constraints in the CDC poll - #39876
Draft
peterdukelarsen wants to merge 4 commits into
Draft
peterdukelarsen wants to merge 4 commits into
peterdukelarsen wants to merge 4 commits into
Conversation
`SchemaChange::hint` renders the recovery for a dropped or altered key as the statements that recreate the table followed by advice to use `EXCLUDE CONSTRAINTS` next time. A source whose `CREATE TABLE .. FROM SOURCE` rejects that option must not give that advice, so the recreate statements are now available on their own as `recreate_hint`, and `hint` builds on it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…poll A dropped UNIQUE constraint on a SQL Server table was invisible to Materialize. SQL Server records only column DDL in `cdc.ddl_history`, which is the only thing the source watched, so after `ALTER TABLE t DROP CONSTRAINT uq` and a duplicate insert the table kept running with its stale key. The optimizer then elided a DISTINCT over that key and returned wrong results while the source reported `running`. The CDC stream now re-reads the PRIMARY KEY and UNIQUE constraints of every tracked table on each poll that found new changes, with one query keyed by capture instance, and yields them as `CdcEvent::Constraints`. Resolving the table through `cdc.change_tables.source_object_id` keeps a renamed table in view, and keying by capture instance gives each of two instances on one table its own copy of the constraints. The replication operator compares the snapshot against the description the export was created with, via `SqlServerTableDesc::check_constraint_compatibility`. Constraints are matched by name and compared by type and column set. Column order is ignored because descriptions migrated from before constraints were recorded hold a composite key's columns in table order while upstream reports key order. A recorded constraint that is gone or altered becomes a definite error built from the shared `mz-source-schema-change` types, so it carries the same diagnosis as Postgres and MySQL, and its `HINT` gives the steps to recreate the table. Added constraints, and constraints that span an excluded column and so were never recorded as keys, are compatible. Column changes stay with the DDL history check, and nullability is not part of this check. Testing: `upstream-schema-changes.td` covers a dropped constraint, an added-then-dropped one, one of several constraints, a constraint over an excluded column, a constraint recreated under the same name over other columns, and a table tracked by two capture instances. The old-syntax `alter-table-after-source.td` asserts the previously TODO'd dropped UNIQUE case. A unit test pins `check_constraint_compatibility`. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The error text alone names the dropped or altered constraint. The recovery hint arrives with EXCLUDE CONSTRAINTS support for SQL Server sources, where the shared hint's advice becomes correct as written. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Five queries built an `IN` list of `@P1, .., @Pn` placeholders and the matching `&dyn ToSql` values, each in its own spelling. `in_list` does it once, without the `as` cast that needed a clippy allowance. Both constraint queries folded rows into `SqlServerTableConstraintRaw` the same way, differing only in the grouping key. `constraints_by_key` does that once, taking the key as a closure. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Fixes https://linear.app/materializeinc/issue/SS-371/sql-server-detect-changes-to-pk-unique-constraints. When a unique constraint is dropped in SQL server, Materialize ignores it and can serve incorrect data.
Description
When processing DDL poke the upstream database to list constraints. If a constrain is dropped, error out.
Verification
Reproduced issue with testdrive test that now passes.
🤖 Generated with Claude Code