Skip to content

Feature/postgres cdc multi schema#4589

Open
ness-david-dedu wants to merge 5 commits into
redpanda-data:mainfrom
ness-david-dedu:feature/postgres_cdc_multi_schema
Open

Feature/postgres cdc multi schema#4589
ness-david-dedu wants to merge 5 commits into
redpanda-data:mainfrom
ness-david-dedu:feature/postgres_cdc_multi_schema

Conversation

@ness-david-dedu

Copy link
Copy Markdown
Contributor
  • Multi-schema replication — schema field now accepts a glob pattern (e.g. tenant_*) in addition to an exact name. All matching schemas are replicated through a single replication slot and publication,
    removing the need for one slot per tenant.
  • commit_ts_ms metadata — transaction commit timestamp (Unix milliseconds) sourced from the WAL BEGIN record; available on all live CDC messages (insert/update/delete), cleared on COMMIT to prevent stale
    state leaking into subsequent transactions.
  • before metadata — JSON-encoded pre-image of the row for update and delete operations. Populated from OldTuple when REPLICA IDENTITY FULL is set on the table; nil for inserts and snapshot reads.
  • pg_schema metadata — schema name of the originating table, useful for per-tenant routing downstream.
  • Bug fix — validateSchemaPattern accepted "" (empty quoted identifier, length 2) due to an off-by-one; corrected to require at least one character inside the quotes.

…t harness

Two-schema (tenant_a, tenant_b) Postgres 16 setup that exercises the
multi-schema CDC pipeline end-to-end. Also adds schema_validation unit
tests that verify invalid patterns are rejected at startup without a DB.

@squiidz squiidz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed after the rebase — the critical issue from the earlier pass is resolved: stream_message.go and replication_message_decoders.go are no longer touched, so the duplicate CommitTs/Before/tuple-decoding reimplementation (which had drifted from what main already shipped in #4554/#4555) is gone. The diff is now correctly scoped to the new schema-glob + pg_schema + publication-batching work layered on top of main's existing commit_ts_ms/before support. 👍

A few remaining items from the original pass, still present:

  1. gofmt: the struct literal in input_pg_stream.go isn't gofmt-clean —

    DBSchemaPattern: schema,
    DBTables:        tables,

    gofmt -w wants DBSchemaPattern: schema, / DBTables: tables, to stay column-aligned with RefreshAuthToken: in the same literal. This will fail a gofmt -l/CI format check as-is.

  2. validateSchemaPattern's quoted-identifier check (input_pg_stream.go) reimplements a shallow prefix/suffix/length check instead of delegating to sanitize.UnquotePostgresIdentifier, which this PR already uses in schema_resolver.go. The existing helper correctly handles escaped double-quotes ("foo""bar"); the new check doesn't.

  3. No CHANGELOG.md entry — recent postgres_cdc PRs (#4554, #4555, ...) all added one under "Unreleased".

  4. tests/current/ (Taskfile/docker-compose/setup.sql/test_config.yaml) reads like a personal manual-test harness — worth confirming this is meant to be committed long-term vs. a scratch dir that should be dropped or renamed to something more descriptive.

  5. Worth a doc callout: schema-pattern resolution happens once at stream construction (resolveSchemas runs at startup only). For the stated multi-tenant use case, a new tenant_c schema created after the pipeline starts won't be picked up until restart. Not a bug, but non-obvious given the motivating use case.

  6. Test coverage gap: no test for the "zero schemas matched" startup-error path (no schemas found matching pattern %q).

None of these block correctness — (1) is the only one that will mechanically fail CI as-is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants