cdc: migrate CDC inputs to canonical config field names (CON-493)#4566
Open
twmb wants to merge 7 commits into
Open
cdc: migrate CDC inputs to canonical config field names (CON-493)#4566twmb wants to merge 7 commits into
twmb wants to merge 7 commits into
Conversation
The CDC Connector Standard (CONTRIBUTING.md §5.3) defines canonical config field names. Several CDC inputs use legacy names. §5.3.2 prescribes a non-breaking migration: add the canonical field, keep the old name accepted, and mark the old one Deprecated(). Add a small shared helper so each connector can resolve a renamed field consistently: prefer the canonical name, fall back to the deprecated name, error when both are set to conflicting values. Both fields must be declared Optional() (no Default()) because benthos hydrates declared defaults into the parsed config, which would otherwise make Contains unable to tell an explicitly-set field from a defaulted one. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Renames snapshot_batch_size to the canonical snapshot_max_batch_size (CONTRIBUTING.md §5.3.1) using the non-breaking pattern from §5.3.2: the old name is still accepted and now marked Deprecated(), the canonical name is preferred, and setting both to conflicting values is rejected. Drops the now-satisfied conformance waiver; the native replication-slot checkpoint waiver remains. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Renames max_parallel_snapshot_objects to the canonical max_parallel_snapshot_tables (CONTRIBUTING.md §5.3.1) via the non-breaking pattern in §5.3.2: the old name is still accepted and now Deprecated(), the canonical name is preferred, and conflicting values are rejected. Removes the now-satisfied salesforce_cdc conformance waiver. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Renames cursor_cache to the canonical checkpoint_cache (CONTRIBUTING.md §5.3.1) via the non-breaking pattern in §5.3.2: the old name is still accepted and now Deprecated(), the canonical name is preferred, and conflicting values are rejected. Drops the now-satisfied checkpoint_cache conformance waiver; the remaining changefeed waivers stay. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…max_batch_size Renames snapshot_parallelism to the canonical max_parallel_snapshot_tables (CONTRIBUTING.md §5.3.1) via the non-breaking pattern in §5.3.2: the old name is still accepted and now Deprecated(), the canonical name is preferred, and conflicting values are rejected. Also adds the canonical snapshot_max_batch_size field controlling the per-batch document count of snapshot reads (previously implicitly tied to read_batch_size), defaulting to 1000 to preserve behavior. Removes the now-satisfied mongodb_cdc conformance waiver. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ot_tables, snapshot_max_batch_size Renames checkpoint_table, snapshot_segments, and snapshot_batch_size to the canonical names from CONTRIBUTING.md §5.3.1 via the non-breaking pattern in §5.3.2: the old names are still accepted and now Deprecated(), the canonical names are preferred, and conflicting values are rejected. The snapshot_mode enum is intentionally left unchanged (§5.3.1 permits an enum where more than two snapshot modes exist; dynamodb has three), so the stream_snapshot waiver remains. Drops the three now-satisfied conformance waivers. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Output of `task docs` after migrating aws_dynamodb_cdc, cockroachdb_changefeed, mongodb_cdc, postgres_cdc, and salesforce_cdc to the canonical CDC config field names. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migrates the CDC inputs to the canonical config field names defined in CONTRIBUTING.md §5.3.1, using the non-breaking pattern from §5.3.2 (precedent:
postgres_cdc'ssnapshot_memory_safety_factor): add the canonical-named field, keep the old name accepted, mark the old oneDeprecated(). Nothing is removed.Mechanism
A shared helper (
internal/cdcfield) resolves each renamed field: prefer the canonical name, fall back to the deprecated name, error if both are set to conflicting values, and apply the default in code. Both the canonical and deprecated fields are declaredOptional()with noDefault()— benthos hydrates declared defaults into the parsed config, so a defaulted field is otherwise indistinguishable (viaContains) from an explicitly-set one, which would defeat conflict detection.Per-connector (canonical ← current)
snapshot_max_batch_size←snapshot_batch_size. Native replication-slot checkpointing left as-is (permanent exemption).checkpoint_cache←checkpoint_table;max_parallel_snapshot_tables←snapshot_segments;snapshot_max_batch_size←snapshot_batch_size. Thesnapshot_modeenum is left unchanged — §5.3.1 permits an enum where more than two snapshot modes genuinely exist, and dynamodb has three.max_parallel_snapshot_tables←snapshot_parallelism; plus a new canonicalsnapshot_max_batch_sizethat now drives the snapshot read batch size (previously implicitly tied toread_batch_size), defaulting to 1000 to preserve behavior.max_parallel_snapshot_tables←max_parallel_snapshot_objects.checkpoint_cache←cursor_cache.Conformance
The now-satisfied entries were removed from
knownNonConformantininternal/plugins/cdctest. Legitimate waivers remain (dynamodbstream_snapshotuses the permittedsnapshot_modeenum; postgrescheckpoint_cacheuses native slot checkpointing; cockroach/gcp_spanner/tigerbeetle untouched).go test -run '^TestCDCConformance$' ./internal/plugins/cdctest/passes.Each connector has a unit test asserting both the old and new field names parse and that conflicting values are rejected.
task docsregenerated the affected component doc pages.🤖 Generated with Claude Code