Skip to content

Support gzip compression in native Parquet writes #4929

Description

@andygrove

What is the problem the feature request solves?

Comet's native Parquet writer supports none, snappy, lz4, and zstd. A write configured with spark.sql.parquet.compression.codec=gzip (or .option("compression", "gzip")) falls back to Spark's writer, since CometDataWritingCommand.getSupportLevel reports Unsupported("Unsupported compression codec: gzip").

gzip is a valid Spark Parquet codec and is widely used, so this is an avoidable loss of native write coverage.

While looking at this, two related gaps in the same code path:

  • parseCompressionCodec does not honor Spark's codec precedence. Spark's ParquetOptions resolves compression > parquet.compression (ParquetOutputFormat.COMPRESSION) > spark.sql.parquet.compression.codec, but Comet checks only the first and third. A write with .option("parquet.compression", "gzip") and the default SQLConf is therefore natively written as snappy, silently substituting the user's requested codec.
  • Spark accepts uncompressed as a spelling of none, but Comet's supported-codec set only lists none, so uncompressed falls back unnecessarily.

Describe the potential solution

Add a Gzip value to the protobuf CompressionCodec enum and map it in the native Parquet writer to Compression::GZIP(GzipLevel::default()) (level 6, matching parquet-mr's zlib default). Fix the codec precedence and accept the uncompressed alias.

Part of #1625.

Additional context

Compression levels remain non-configurable (gzip and zstd use fixed defaults), so parquet-mr's parquet.compression.codec.{gzip,zstd}.level configs are still ignored. That is out of scope here.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions