Enable the split-operator plan and native Iceberg writes by default #5644

What is the problem the feature request solves?

#6664 made Comet's Iceberg write path the default on every Spark version, and #6962 turned the default back off for 1.2.0, which ships native Iceberg writes as a supported, opt-in feature. spark.comet.write.iceberg.enabled (renamed from spark.comet.iceberg.write.enabled by #5306) is the one setting: it defaults to false, and when it is on it plans both the writer/committer split from #4658 and the iceberg-rust data-file writer from #5361. spark.comet.write.iceberg.splitOperator.enabled is a testing setting that plans the split operator with the native writer off. The split plan changes only the plan shape. The native writer writes the data files of each write its eligibility gate accepts; every other write falls back to iceberg-java's writer, and iceberg-java commits every write. With the setting off, Spark plans its own write operator, as in Comet 1.1.0.

This issue defines what has to be true before a release ships with the default on. Once the 1.2.0 branch is cut, the default can return to true on main, so that it runs as the default for a full release cycle before 1.3.0 (#6962). The 1.2.0 milestone is due on 2026-10-23 (#6550). Everything that had to be fixed before the default flipped is fixed (listed below). What is left is evidence, which CI and the nightly runs keep gathering with the default off: every Iceberg Spark test diff under dev/diffs/iceberg/ and both Iceberg write benchmarks set spark.comet.write.iceberg.enabled=true themselves.

Describe the potential solution

Before a release ships with the default on

Evidence gathered while CI and the nightly runs exercise the native writer:

• Ten nightly runs in which the Iceberg jobs ran with the native writer on, and no failure that only the native writer causes. The first is the 2026-10-14 nightly. A nightly runs only the jobs its diff from the last green nightly routes to, so a nightly that skips the Iceberg jobs does not count. Iceberg 1.8, 1.9 and 1.10 run nightly, and 1.11 in the merge queue.
• Benchmark results (CometIcebergWriteBenchmark, Add a native Iceberg write benchmark #5647) show no regression against iceberg-java for unpartitioned, clustered, fanout and copy-on-write delete writes. On a local filesystem the native writer is faster in all four: 2.1x unpartitioned, 1.4x clustered, 1.9x fanout and 1.4x for a copy-on-write DELETE (table). Object-store latency is not measured. There the native path also makes two small reads per written file to read its footer back (Hand natively written Iceberg files' Parquet footers to the JVM instead of reading them back #6772).
• Each fallback reason in the Iceberg CI write report is either intended or tracked. In the Iceberg 1.11 jobs of the merge-queue run that merged feat: enable native Iceberg writes by default #6664 (2026-10-13), 7,175 of the 13,677 writes ran natively (52.5%). A write can have several reasons, so the counts overlap.
  • Intended: Avro and ORC tables (2,458), format-version 4 (581), and the test suites' own FileIO and file system classes (128)
  • Tracked: merge-on-read WriteDelta (1,477, Support Native Iceberg MOR write #6240), format-version 3 copy-on-write and compaction writes that carry row lineage (215, Native Iceberg writes: write row lineage for copy-on-write DML and compaction on format-version 3 tables #6669), and float or double identity partitions (18, PR feat: bump iceberg-rust and write float and double identity partitions natively #6851)
  • Waiting on a Decide keep-or-lift for each remaining native Iceberg write eligibility restriction #5643 decision: variant (77), uuid (11) and unknown (4) columns, and encryption.* properties (17)
  • Not yet classified: input that is not in Arrow format (1,058), writes with no reason recorded (147), "IcebergWrite is not supported" (63), "could not determine the table format-version" (49), and 199 writes that Spark planned without the split operator (189 AppendData, 10 streaming WriteToDataSourceV2)
• Manifest metrics parity asserted per data type on all four pinned Iceberg versions (1.5.2, 1.8.1, 1.10.0, 1.11.0)
• Decide keep-or-lift for each remaining native Iceberg write eligibility restriction #5643 Every remaining eligibility restriction has a keep-or-lift decision (bloom filters: PR feat: support bloom filters in native Iceberg writes #5724; float/double identity partitions: PR feat: bump iceberg-rust and write float and double identity partitions natively #6851)

Spark 3.4 gets the same defaults. No Iceberg Spark test job covers Iceberg 1.5.2, which the 3.4 profile pins, so on 3.4 the evidence is Comet's own suites.

Not blocking the default

• revertToSpark erases CometIcebergWriteExec / CometNativeWriteExec because originalPlan is the node's own child #5719 revertToSpark keeps the write node (PR fix: restore Spark write execs when reverting transition-heavy stages #5957). It is reached only with spark.comet.exec.transitionRevert.enabled, which is off by default.
• Failure-injection tests for native Iceberg writes #5646 Failure-injection tests that check storage state (PR test: cover speculation and executor loss in multi-task native Iceberg writes #6601 covers speculation and executor loss)

Fixed before the defaults flipped

With the native writer on by default, each of these would have reached every user who writes Iceberg tables.

Split plan:

• Iceberg write split-operator plan: CI failures when enabled by default #5259 The four CI failure buckets seen with the split plan on by default are fixed
• IcebergWriteStrategy ignores spark.comet.enabled, so the split-operator write plan survives the kill switch #6142 IcebergWriteStrategy respects spark.comet.enabled, so disabling Comet restores Spark's plan
• IcebergCommitExec rethrows the raw cause when the abort fails, where Spark wraps it in writingJobFailedError #6143 Job and commit failures surface as the same exception types as Spark's own write
• Preserve catalog transactions for split Iceberg writes on Spark 4.2 #6586 Catalog transactions are committed on Spark 4.2 (TransactionalExec)

Native writer correctness:

• Native Iceberg writes use an iceberg-rust revision missing partition-path escaping #5636, Native Iceberg writes drop Hadoop GCS configuration for HadoopFileIO + gs:// #5637, Native Iceberg write merges -0.0 and 0.0 rows into one float/double identity partition #6138, Native Iceberg write gate reads hdfs:/path as file, so the write fails natively instead of falling back #6140, Native Iceberg write panics or writes a NULL partition value for dates and timestamps beyond year 262143 #6145, docs: fix stale and missing native write details in iceberg-writes.md and iceberg.md #6147
• Native Iceberg write silently drops S3 settings it cannot honour (credentials provider, SSE, ACL, tags, remote signing) instead of falling back #6139 S3 settings the native writer cannot honour fall back instead of being dropped (PR fix: gate unsupported Iceberg S3 write settings #6441)
• Native Iceberg write fails on a V1 spec that mixes a live field with a void field whose source column was dropped #6141 A V1 spec that mixes a live field with a void field whose source column was dropped falls back, so the write fails with iceberg-java's own error (PR fix: decline native Iceberg writes through a void partition field whose source column was dropped #6677)
• Native Iceberg write over-counts NaNs for nested float fields when the input batch is already sliced #6146 and Native Iceberg writer counts NaNs under NULL structs and NULL list or map entries #6562 NaN counts for nested float fields (PRs fix: compact already-sliced batches before the native Iceberg writer counts NaNs #6238 and fix: drop values under NULL parents before the native Iceberg writer counts NaNs #6658)

Failure handling:

• Native Iceberg write: clean up task-attempt data files on task failure #5618, Remove orphaned files in comet iceberg split writers #5277, Native Iceberg Write: close cleanup ownership gap during JVM handoff #6581
• Spark errors after the first batch of a JVM input stream reach the user as CometNativeException #6234 A Spark error after the first input batch reaches the user as the exception Spark threw (PR fix: rethrow the exception a JVM input throws instead of a CometNativeException #6243)

Memory and performance:

• Account the native Iceberg writer's buffers in Comet's memory pool #5648 and Charge native write buffers to the shared off-heap memory pool #6115 The writer's buffers, including the fanout writer's per-partition buffers, are charged to Comet's memory pool (PR feat: charge native write buffers to the task's memory pool #6247)
• Native Iceberg writer keeps a dictionary page for high-cardinality columns where iceberg-java writes none #6114 The native writer drops the dictionary of a high-cardinality column the way iceberg-java does, so selective reads no longer fetch a ~2 MB dictionary page per column (PR perf: choose dictionary encoding like iceberg-java in the native Iceberg writer #6241)

Coverage and evidence:

• Support Iceberg's system functions (bucket, truncate, years/months/days/hours) as native expressions #5635 The default hash-distribution write is native end to end
• Make native Iceberg write coverage visible in the Iceberg Spark test CI jobs #6148 The Iceberg CI jobs report which writer ran each write
• Iceberg's Spark test suites run with both flags on: the split plan since feat: Optionally split the Iceberg V2 write operator into distinct writer and committer operations #4658, the native writer since test: run the Iceberg Spark tests with the native Iceberg writer enabled #5677 (2026-09-22). Iceberg 1.8, 1.9 and 1.10 run nightly, 1.11 in the merge queue. From 2026-09-16 to 2026-10-04, 19 nightly runs of 25 Iceberg jobs each had one failure, the upstream Iceberg 1.9 partial-progress flake (Flaky test: TestRewriteDataFilesAction.testParallelPartialProgressWithMaxFailedCommitsLargerThanTotalFileGroup() iceberg#12889). In the last 30 merge-queue runs, the one Iceberg job that failed stopped before any test ran.
• Comet's own suites pass with both defaults on, on every Spark profile (feat: enable native Iceberg writes by default #6664's final CI run, with run-all-spark-profiles and run-iceberg-tests, passed)

Done in #6664

• spark.comet.write.iceberg.enabled moves from CATEGORY_TESTING to CATEGORY_EXEC and switches both layers, and the docs that describe the flags as off by default are updated: user-guide/latest/iceberg-writes.md, iceberg.md, operators.md, contributor-guide/iceberg-writes.md and the Iceberg write review skill
• Release note: a 1.2.0 upgrade-guide entry names the setting and how to turn it off. Iceberg write plans show IcebergCommit over IcebergWrite (or CometIcebergWrite) in place of Spark's AppendData, OverwriteByExpression, OverwritePartitionsDynamic and ReplaceData exec nodes. The split happens at physical planning, so analyzed and optimized logical plans are unchanged, and listeners that read logical plans (lineage tools) are unaffected. Code that matches Spark's physical write nodes will not find them.

Done in #6962

• spark.comet.write.iceberg.enabled defaults to false again, so 1.2.0 ships the native writer as a supported, opt-in feature. The user guide, the FAQ, the contributor guide, the roadmap and the Iceberg write review skill describe it as opt-in, and the 1.2.0 upgrade-guide entry covers the move from 1.1.0's testing settings to the one supported setting.

Additional context

Part of #5649. Refreshed on 2026-10-13, after #6664 merged, and on 2026-10-16, after #6962 merged. Earlier versions folded in the original list from 2026-09-02 and the issues the 2026-09-23 audit added (#6138 to #6148). Related: #4658, #5298, #5361.