Skip to content

docs: expand tuning guide with performance and memory configs#4908

Open
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:docs/tuning-guide-1.0
Open

docs: expand tuning guide with performance and memory configs#4908
andygrove wants to merge 1 commit into
apache:mainfrom
andygrove:docs/tuning-guide-1.0

Conversation

@andygrove

Copy link
Copy Markdown
Member

Summary

Audited all CometConf entries in preparation for the 1.0 release and expanded the Tuning Guide with the user-facing performance and memory knobs that were previously undocumented.

New sections in tuning.md:

  • Batch Sizespark.comet.batchSize (memory vs. throughput tradeoff) and its shuffle-writer cousin spark.comet.columnar.shuffle.batch.size
  • Limiting Spill Disk Usagespark.comet.maxTempDirectorySize (100 GB default)
  • Parquet Reader Tuning — parallel I/O thread pool, range coalescing, filter pushdown / late materialization
  • Iceberg Scan Tuningspark.comet.scan.icebergNative.dataFileConcurrencyLimit
  • Reducing Row/Columnar Conversion Overheadspark.comet.exec.transitionRevert.{enabled,maxTransitions}
  • Metrics Overhead — why spark.comet.metrics.enabled is off by default
  • Expanded Shuffle Compression with codec / zstd level knobs

Also:

  • Fixed an incorrect claim that Comet overrides Spark LZ4 shuffle compression with ZSTD (the default codec is lz4).
  • Moved spark.comet.maxTempDirectorySize from the exec category to tuning so it renders in the Memory & Tuning table in configs.md, and expanded its doc string.

Test plan

  • Docs build renders new sections correctly
  • configs.md still generates cleanly (moved config appears in Memory & Tuning table)

Audited all CometConf entries and added documentation for user-facing
performance and memory knobs that were undocumented in the tuning guide:
batch size, spill disk cap, Parquet parallel I/O and filter pushdown,
Iceberg data-file concurrency, shuffle compression codec/level,
row/columnar transition revert, and metrics overhead. Also fixed an
incorrect claim about the default shuffle compression codec (LZ4, not
ZSTD) and moved spark.comet.maxTempDirectorySize into the tuning
category so it renders in the Memory & Tuning table.
@andygrove andygrove marked this pull request as ready for review July 13, 2026 17:12
@andygrove andygrove changed the title docs: expand tuning guide with performance and memory configs for 1.0 docs: expand tuning guide with performance and memory configs Jul 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant