What is the problem the feature request solves?
Comet's native Parquet writer supports none, snappy, lz4, and zstd. A write configured with spark.sql.parquet.compression.codec=gzip (or .option("compression", "gzip")) falls back to Spark's writer, since CometDataWritingCommand.getSupportLevel reports Unsupported("Unsupported compression codec: gzip").
gzip is a valid Spark Parquet codec and is widely used, so this is an avoidable loss of native write coverage.
While looking at this, two related gaps in the same code path:
parseCompressionCodec does not honor Spark's codec precedence. Spark's ParquetOptions resolves compression > parquet.compression (ParquetOutputFormat.COMPRESSION) > spark.sql.parquet.compression.codec, but Comet checks only the first and third. A write with .option("parquet.compression", "gzip") and the default SQLConf is therefore natively written as snappy, silently substituting the user's requested codec.
- Spark accepts
uncompressed as a spelling of none, but Comet's supported-codec set only lists none, so uncompressed falls back unnecessarily.
Describe the potential solution
Add a Gzip value to the protobuf CompressionCodec enum and map it in the native Parquet writer to Compression::GZIP(GzipLevel::default()) (level 6, matching parquet-mr's zlib default). Fix the codec precedence and accept the uncompressed alias.
Part of #1625.
Additional context
Compression levels remain non-configurable (gzip and zstd use fixed defaults), so parquet-mr's parquet.compression.codec.{gzip,zstd}.level configs are still ignored. That is out of scope here.
What is the problem the feature request solves?
Comet's native Parquet writer supports
none,snappy,lz4, andzstd. A write configured withspark.sql.parquet.compression.codec=gzip(or.option("compression", "gzip")) falls back to Spark's writer, sinceCometDataWritingCommand.getSupportLevelreportsUnsupported("Unsupported compression codec: gzip").gzip is a valid Spark Parquet codec and is widely used, so this is an avoidable loss of native write coverage.
While looking at this, two related gaps in the same code path:
parseCompressionCodecdoes not honor Spark's codec precedence. Spark'sParquetOptionsresolvescompression>parquet.compression(ParquetOutputFormat.COMPRESSION) >spark.sql.parquet.compression.codec, but Comet checks only the first and third. A write with.option("parquet.compression", "gzip")and the default SQLConf is therefore natively written as snappy, silently substituting the user's requested codec.uncompressedas a spelling ofnone, but Comet's supported-codec set only listsnone, souncompressedfalls back unnecessarily.Describe the potential solution
Add a
Gzipvalue to the protobufCompressionCodecenum and map it in the native Parquet writer toCompression::GZIP(GzipLevel::default())(level 6, matching parquet-mr's zlib default). Fix the codec precedence and accept theuncompressedalias.Part of #1625.
Additional context
Compression levels remain non-configurable (gzip and zstd use fixed defaults), so parquet-mr's
parquet.compression.codec.{gzip,zstd}.levelconfigs are still ignored. That is out of scope here.