Skip to content

chore: Support native writer Overwrite and ErrorIfExists modes#4946

Merged
comphead merged 2 commits into
apache:mainfrom
comphead:writer_modes
Jul 16, 2026
Merged

chore: Support native writer Overwrite and ErrorIfExists modes#4946
comphead merged 2 commits into
apache:mainfrom
comphead:writer_modes

Conversation

@comphead

@comphead comphead commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Fixes Comet's native Parquet writer honoring SaveMode. Because CometExecRule replaces DataWritingCommandExec with CometNativeWriteExec, Spark's InsertIntoHadoopFsRelationCommand.run() — where the
    path-exists check, delete-before-Overwrite, and Ignore short-circuit live — never runs. Overwrite against a populated target was silently acting like Append.
  • Ports the non-partitioned/non-catalog branch of Spark 3's doInsertion logic into CometNativeWriteExec.doExecuteColumnar(): ErrorIfExists + exists → throw, Overwrite + exists → delete, Ignore +
    exists → skip, Append → proceed. SaveMode is now threaded through CometDataWritingCommand.
  • Adds 10 tests in CometParquetWriterSuite covering each mode against empty / populated / empty-DataFrame targets, plus a materializeAsCometSource helper (inline Seq.toDF produces a LocalTableScan
    that trips requiresNativeChildren and silently falls back to Spark's writer).

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @comphead.

I did not try and do a comprehensive AI review on this one because this feature is marked as experimental and disabled by default. We can keep iterating and do a full audit later, IMO.

@comphead

Copy link
Copy Markdown
Contributor Author

Thanks @andygrove we gonna improve it as we test

@comphead comphead merged commit 2fd3960 into apache:main Jul 16, 2026
70 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants