changefeedccl: support top-level compression option for Kafka sinks#164584
Open
varundeepsaini wants to merge 1 commit intocockroachdb:masterfrom
Open
changefeedccl: support top-level compression option for Kafka sinks#164584varundeepsaini wants to merge 1 commit intocockroachdb:masterfrom
varundeepsaini wants to merge 1 commit intocockroachdb:masterfrom
Conversation
Release note (enterprise change): Kafka changefeeds now accept the top-level `compression` option (e.g. `WITH compression='gzip'`), matching the syntax already supported by other sink types. Signed-off-by: Varun Deep Saini <varun.23bcs10048@ms.sst.scaler.com>
Contributor
|
Merging to
|
|
Thank you for contributing to CockroachDB. Please ensure you have followed the guidelines for creating a PR. My owl senses detect your PR is good for review. Please keep an eye out for any test failures in CI. 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
Member
|
This change is |
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #113495
Previously, the compression changefeed option was incompatible with Kafka
sinks. Users had to configure compression through kafka_sink_config JSON,
which was inconsistent with other sinks (cloud storage, webhook) that
supported the top-level compression option directly.
This PR adds support for the compression option on Kafka sinks,
accepting all Kafka-supported codecs: gzip, snappy, lz4, and zstd.
When both the top-level compression option and kafka_sink_config.Compression
are set, they must match or an error is returned to avoid ambiguous
configuration. The change applies to both the v1 (sarama) and v2 (kgo)
Kafka sink implementations.
Additionally, the compression option validation is changed from a
fixed enum (gzip, zstd) to a free-form string, since valid compression
codecs are sink-dependent (e.g., Kafka supports snappy and lz4 in
addition to gzip and zstd). Each sink now validates the compression
value it receives.