Skip to content

proto: layered DataSource / FileSource / DataSink nodes (precursor to #26555, #26807) - #26824

Open
adriangb wants to merge 4 commits into
apache:mainfrom
adriangb:proto-datasource-node-layer
Open

adriangb wants to merge 4 commits into
apache:mainfrom
adriangb:proto-datasource-node-layer

Conversation

@adriangb

@adriangb adriangb commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

The DataSource, FileSource and DataSink proto hooks (added in 55.0.0) return a full PhysicalPlanNode, which is the Exec wrapper. Decode in #26555 / #26807 works at the source level. So:

Problem Result
A source must build its own wrapper Public helpers (extension_data_source_node); a source can return any plan variant
Built-in sources use their own plan variants Two ways to serialize a source (thread)
No file-source level A custom FileSource needs a marker DataSource
No source level on the wire An unregistered reader gives an unclear error, or decodes the wrong source

This PR adds a proto layer that mirrors the traits. Then #26555 / #26807 only add a decoder for the extension arms.

Only the scan / sink variants of PhysicalPlanNode are shown. The other ExecutionPlan variants (projection = 4, filter = 12, ...) do not change.

Before:

PhysicalPlanNode
├─ ...                 (other ExecutionPlan variants)
├─ parquet_scan = 1
├─ csv_scan     = 2
├─ avro_scan    = 17   (deprecated)
├─ json_scan    = 31   (deprecated)
├─ memory_scan  = 35   (deprecated)
├─ arrow_scan   = 38   (deprecated)
├─ json_sink    = 24   (deprecated)
├─ csv_sink     = 28   (deprecated)
└─ parquet_sink = 29   (deprecated)

After:

PhysicalPlanNode
├─ ...                 (other ExecutionPlan variants)
├─ parquet_scan = 1 ... parquet_sink = 29   (deprecated: decode only)
├─ data_source = 42: DataSourceExecNode
│  └─ DataSourceNode (oneof)
│     ├─ file: FileScanNode
│     │  ├─ base: FileScanExecConf
│     │  └─ FileSourceNode (oneof)
│     │     ├─ parquet / csv / json / avro / arrow
│     │     └─ extension {name, payload}
│     ├─ memory
│     └─ extension {name, payload}
└─ data_sink = 43: DataSinkExecNode {input, sort_order}
   └─ DataSinkNode (oneof)
      ├─ json / csv / parquet
      └─ extension {name, payload}

Timing: land before 56.0.0 (#24461), together with #26555 / #26807. Then the hooks added in 55.0.0 change only once, through deprecation.

What changes are included in this PR?

Area Change
Proto New messages (see above). The new format messages keep the field tags of the legacy scan nodes, so the legacy → new translation only moves fields.
Encode Built-ins always write the new form.
Decode (legacy) Each deprecated variant decodes in one call through the new decoders (decode_file_scan, build_data_sink), by reference, with no subtree copy. Format fields map through From<&XScanExecNode> for XSourceNode in datafusion-proto-models/src/physical_plan_compat.rs: one mapping per format, exhaustive destructure. The deprecated XSource::try_from_proto helpers use the same impls.
New hooks (default Ok(None)) DataSource::try_to_proto_source → DataSourceNode
FileSource::try_to_proto_file_source → FileSourceNode
DataSink::try_to_proto_sink → DataSinkNode
Deprecated (not removed) The try_to_proto hooks added in 55.0.0, and the old public decode functions. They are otherwise unchanged: the hook defaults still return None, and built-ins implement only the new hooks.
Encode order new hook → deprecated hook → PhysicalExtensionCodec
extension arms Decode returns No decoder for extension DataSource '{name}'. The registry is in #26555 / #26807.

Suggested shape for #26555 after this PR:

pub trait ExtensionDataSourceFromProto: DataSource + Sized {
    const NAME: &'static str;
    fn try_from_proto(payload: &[u8], ctx: &ExecutionPlanDecodeCtx<'_>) -> Result<Arc<dyn DataSource>>;
    fn register_proto(registry: &mut ProtoDecoderRegistry) -> Result<()> { /* provided */ }
}

What is the testing strategy for this PR?

datafusion/proto/tests/cases/plans/deprecated_data_source_proto_serialization.rs:

Test Checks
built_ins_encode_layered_and_legacy_variants_decode Each built-in round-trips in the new form; its legacy variant decodes to the same plan and re-encodes byte-identical to the new form
missing_fields_give_clear_errors, extension_nodes_without_decoder_error Error text

Existing scan and sink tests in sources.rs / sinks.rs now read the layered form.

Feature flags: cargo check -p datafusion-proto passes with no default features, avro, default and all features. -p datafusion-datasource and -p datafusion pass with no default features. cargo test -p datafusion-proto --no-default-features does not compile, on main too: existing tests import Parquet types without a gate.

Are there any user-facing changes?

  • Deprecations: the try_to_proto hooks (added in 55.0.0) on DataSource, FileSource and DataSink. Existing implementations still work. Migration notes are in upgrading/56.0.0.md.
  • Wire: a 56.0.0 writer produces plans that a 55.x reader cannot decode. A 56.0.0 reader decodes 55.x plans. (Plans are not guaranteed compatible across versions.)

Open points

  • #[deprecated] on a trait method gives no warning to implementers who override it.
  • The generated code was produced with protox because protoc was not available. Run datafusion/proto-models/regen.sh to confirm.
  • Commits: (1) proto messages and generated code, (2) hooks and encode/decode, (3) tests, (4) docs. Commit 1 alone does not build datafusion-proto (non-exhaustive match on the new variants). This is OK for a squash merge.

🤖 Generated with Claude Code

@github-actions github-actions Bot added documentation Improvements or additions to documentation proto Related to proto crate datasource Changes to the datasource crate labels Oct 27, 2026
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details
     Cloning 985bbbf60e14644d8945dce9342f6dd20a428f1a
    Building datafusion-datasource v55.1.0 (current)
       Built [  39.073s] (current)
     Parsing datafusion-datasource v55.1.0 (current)
      Parsed [   0.029s] (current)
    Building datafusion-datasource v55.1.0 (baseline)
       Built [  14.993s] (baseline)
     Parsing datafusion-datasource v55.1.0 (baseline)
      Parsed [   0.036s] (baseline)
    Checking datafusion-datasource v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.308s] 229 checks: 227 pass, 2 fail, 0 warn, 31 skip

--- failure trait_method_marked_deprecated: trait method #[deprecated] added ---

Description:
A trait method is now #[deprecated]. Downstream crates will get a compiler warning when using this method.
        ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/trait_method_marked_deprecated.ron

Failed in:
  method try_to_proto in trait datafusion_datasource::file::FileSource in /home/runner/work/datafusion/datafusion/datafusion/datasource/src/file.rs:66
  method try_to_proto in trait datafusion_datasource::sink::DataSink in /home/runner/work/datafusion/datafusion/datafusion/datasource/src/sink.rs:50
  method try_to_proto in trait datafusion_datasource::source::DataSource in /home/runner/work/datafusion/datafusion/datafusion/datasource/src/source.rs:128

--- failure type_method_marked_deprecated: type method #[deprecated] added ---

Description:
A type method is now #[deprecated]. Downstream crates will get a compiler warning when using this method.
        ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/type_method_marked_deprecated.ron

Failed in:
  method datafusion_datasource::memory::MemorySourceConfig::try_from_proto in /home/runner/work/datafusion/datafusion/datafusion/datasource/src/memory.rs:702

     Summary semver requires new minor version: 0 major and 2 minor checks failed
    Finished [  56.716s] datafusion-datasource
    Building datafusion-datasource-arrow v55.1.0 (current)
       Built [  15.250s] (current)
     Parsing datafusion-datasource-arrow v55.1.0 (current)
      Parsed [   0.007s] (current)
    Building datafusion-datasource-arrow v55.1.0 (baseline)
       Built [  14.855s] (baseline)
     Parsing datafusion-datasource-arrow v55.1.0 (baseline)
      Parsed [   0.007s] (baseline)
    Checking datafusion-datasource-arrow v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.084s] 229 checks: 228 pass, 1 fail, 0 warn, 31 skip

--- failure type_method_marked_deprecated: type method #[deprecated] added ---

Description:
A type method is now #[deprecated]. Downstream crates will get a compiler warning when using this method.
        ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/type_method_marked_deprecated.ron

Failed in:
  method datafusion_datasource_arrow::source::ArrowSource::try_from_proto in /home/runner/work/datafusion/datafusion/datafusion/datasource-arrow/src/source.rs:451

     Summary semver requires new minor version: 0 major and 1 minor checks failed
    Finished [  38.349s] datafusion-datasource-arrow
    Building datafusion-datasource-avro v55.1.0 (current)
       Built [  16.155s] (current)
     Parsing datafusion-datasource-avro v55.1.0 (current)
      Parsed [   0.006s] (current)
    Building datafusion-datasource-avro v55.1.0 (baseline)
       Built [  15.055s] (baseline)
     Parsing datafusion-datasource-avro v55.1.0 (baseline)
      Parsed [   0.006s] (baseline)
    Checking datafusion-datasource-avro v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.085s] 229 checks: 228 pass, 1 fail, 0 warn, 31 skip

--- failure type_method_marked_deprecated: type method #[deprecated] added ---

Description:
A type method is now #[deprecated]. Downstream crates will get a compiler warning when using this method.
        ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/type_method_marked_deprecated.ron

Failed in:
  method datafusion_datasource_avro::source::AvroSource::try_from_proto in /home/runner/work/datafusion/datafusion/datafusion/datasource-avro/src/source.rs:215

     Summary semver requires new minor version: 0 major and 1 minor checks failed
    Finished [  32.519s] datafusion-datasource-avro
    Building datafusion-datasource-csv v55.1.0 (current)
       Built [   0.403s] (current)
     Parsing datafusion-datasource-csv v55.1.0 (current)
      Parsed [   0.006s] (current)
    Building datafusion-datasource-csv v55.1.0 (baseline)
       Built [   0.409s] (baseline)
     Parsing datafusion-datasource-csv v55.1.0 (baseline)
      Parsed [   0.007s] (baseline)
    Checking datafusion-datasource-csv v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.116s] 229 checks: 228 pass, 1 fail, 0 warn, 31 skip

--- failure type_method_marked_deprecated: type method #[deprecated] added ---

Description:
A type method is now #[deprecated]. Downstream crates will get a compiler warning when using this method.
        ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/type_method_marked_deprecated.ron

Failed in:
  method datafusion_datasource_csv::file_format::CsvSink::try_from_proto in /home/runner/work/datafusion/datafusion/datafusion/datasource-csv/src/file_format.rs:928
  method datafusion_datasource_csv::CsvSink::try_from_proto in /home/runner/work/datafusion/datafusion/datafusion/datasource-csv/src/file_format.rs:928
  method datafusion_datasource_csv::source::CsvSource::try_from_proto in /home/runner/work/datafusion/datafusion/datafusion/datasource-csv/src/source.rs:609

     Summary semver requires new minor version: 0 major and 1 minor checks failed
    Finished [   1.851s] datafusion-datasource-csv
    Building datafusion-datasource-json v55.1.0 (current)
       Built [   0.588s] (current)
     Parsing datafusion-datasource-json v55.1.0 (current)
      Parsed [   0.008s] (current)
    Building datafusion-datasource-json v55.1.0 (baseline)
       Built [   0.416s] (baseline)
     Parsing datafusion-datasource-json v55.1.0 (baseline)
      Parsed [   0.008s] (baseline)
    Checking datafusion-datasource-json v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.104s] 229 checks: 228 pass, 1 fail, 0 warn, 31 skip

--- failure type_method_marked_deprecated: type method #[deprecated] added ---

Description:
A type method is now #[deprecated]. Downstream crates will get a compiler warning when using this method.
        ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/type_method_marked_deprecated.ron

Failed in:
  method datafusion_datasource_json::source::JsonSource::try_from_proto in /home/runner/work/datafusion/datafusion/datafusion/datasource-json/src/source.rs:291
  method datafusion_datasource_json::file_format::JsonSink::try_from_proto in /home/runner/work/datafusion/datafusion/datafusion/datasource-json/src/file_format.rs:569
  method datafusion_datasource_json::JsonSink::try_from_proto in /home/runner/work/datafusion/datafusion/datafusion/datasource-json/src/file_format.rs:569

     Summary semver requires new minor version: 0 major and 1 minor checks failed
    Finished [   1.914s] datafusion-datasource-json
    Building datafusion-datasource-parquet v55.1.0 (current)
       Built [  21.222s] (current)
     Parsing datafusion-datasource-parquet v55.1.0 (current)
      Parsed [   0.030s] (current)
    Building datafusion-datasource-parquet v55.1.0 (baseline)
       Built [  15.911s] (baseline)
     Parsing datafusion-datasource-parquet v55.1.0 (baseline)
      Parsed [   0.032s] (baseline)
    Checking datafusion-datasource-parquet v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.185s] 229 checks: 228 pass, 1 fail, 0 warn, 31 skip

--- failure type_method_marked_deprecated: type method #[deprecated] added ---

Description:
A type method is now #[deprecated]. Downstream crates will get a compiler warning when using this method.
        ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/type_method_marked_deprecated.ron

Failed in:
  method datafusion_datasource_parquet::source::ParquetSource::try_from_proto in /home/runner/work/datafusion/datafusion/datafusion/datasource-parquet/src/source.rs:1159
  method datafusion_datasource_parquet::file_format::ParquetSink::try_from_proto in /home/runner/work/datafusion/datafusion/datafusion/datasource-parquet/src/sink.rs:554
  method datafusion_datasource_parquet::ParquetSink::try_from_proto in /home/runner/work/datafusion/datafusion/datafusion/datasource-parquet/src/sink.rs:554

     Summary semver requires new minor version: 0 major and 1 minor checks failed
    Finished [  38.356s] datafusion-datasource-parquet
    Building datafusion-proto v55.1.0 (current)
       Built [  22.758s] (current)
     Parsing datafusion-proto v55.1.0 (current)
      Parsed [   0.012s] (current)
    Building datafusion-proto v55.1.0 (baseline)
       Built [  18.807s] (baseline)
     Parsing datafusion-proto v55.1.0 (baseline)
      Parsed [   0.012s] (baseline)
    Checking datafusion-proto v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   0.130s] 229 checks: 229 pass, 31 skip
     Summary no semver update required
    Finished [  42.767s] datafusion-proto
    Building datafusion-proto-models v55.1.0 (current)
       Built [  17.772s] (current)
     Parsing datafusion-proto-models v55.1.0 (current)
      Parsed [   0.145s] (current)
    Building datafusion-proto-models v55.1.0 (baseline)
       Built [   5.226s] (baseline)
     Parsing datafusion-proto-models v55.1.0 (baseline)
      Parsed [   0.143s] (baseline)
    Checking datafusion-proto-models v55.1.0 -> v55.1.0 (no change; assume patch)
     Checked [   2.504s] 229 checks: 228 pass, 1 fail, 0 warn, 31 skip

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/enum_variant_added.ron

Failed in:
  variant PhysicalPlanType:DataSource in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1501
  variant PhysicalPlanType:DataSink in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1503
  variant PhysicalPlanType:DataSource in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1501
  variant PhysicalPlanType:DataSink in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1503

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  26.861s] datafusion-proto-models

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Deprecated encoders now emit layered nodes that their paired decoders reject, breaking existing source and sink round trips and requiring decoder compatibility fixes and tests.

1 open issue
What changed in this PR

This PR introduces layered protobuf nodes that mirror DataFusion’s source and sink traits, preparing for the per-type decoder registries in #26555 and #26807.

Changes:

  • Adds source, file-source, and sink messages and migrates built-in encoding to the layered format.
  • Retains legacy decoding and deprecated hooks while centralizing compatibility conversions.
  • Adds compatibility tests and documents migration and reader/writer version constraints.
File Description
docs/​source/​library-user-guide/​upgrading/​56.0.0.md Documents deprecated hooks and wire-format migration.
datafusion/​proto/​tests/​cases/​plans/​sources.rs Updates source assertions for layered nodes.
datafusion/​proto/​tests/​cases/​plans/​sinks.rs Updates sink assertions for layered nodes.
datafusion/​proto/​tests/​cases/​plans/​mod.rs Registers compatibility tests.
datafusion/​proto/​tests/​cases/​plans/​legacy.rs Tests layered encoding, legacy decoding, and fallback behavior.
datafusion/​proto/​src/​physical_plan/​mod.rs Dispatches layered and legacy source/sink decoding.
datafusion/​proto-models/​src/​physical_plan_compat.rs Maps legacy scan fields into source messages.
datafusion/​proto-models/​src/​lib.rs Exposes compatibility conversions.
datafusion/​proto-models/​src/​generated/​prost.rs Adds generated layered protobuf types.
datafusion/​proto-models/​proto/​datafusion.proto Defines layered messages and retains legacy variants.
datafusion/​datasource/​src/​source.rs Adds source-level encoding and wrapper construction.
datafusion/​datasource/​src/​sink.rs Adds sink-level encoding and wrapper construction.
datafusion/​datasource/​src/​memory.rs Migrates memory-source serialization.
datafusion/​datasource/​src/​file.rs Adds file-source encoding with deprecated-hook forwarding.
datafusion/​datasource/​src/​file_scan_config/​mod.rs Combines shared scan configuration with source messages.
datafusion/​datasource-parquet/​src/​source.rs Migrates Parquet source serialization.
datafusion/​datasource-parquet/​src/​sink.rs Migrates Parquet sink encoding.
datafusion/​datasource-json/​src/​source.rs Migrates JSON source serialization.
datafusion/​datasource-json/​src/​file_format.rs Migrates JSON sink encoding.
datafusion/​datasource-csv/​src/​source.rs Migrates CSV source serialization.
datafusion/​datasource-csv/​src/​file_format.rs Migrates CSV sink encoding.
datafusion/​datasource-avro/​src/​source.rs Migrates Avro source serialization.
datafusion/​datasource-arrow/​src/​source.rs Migrates Arrow source serialization.

Review effort: Balanced

Comment thread datafusion/datasource/src/source.rs Outdated
adriangb and others added 2 commits October 27, 2026 18:09
Adds data_source = 42 and data_sink = 43 to PhysicalPlanNode, with
DataSourceNode, FileSourceNode and DataSinkNode oneofs that each have an
extension {name, payload} arm. Generated code only; datafusion-proto
handles the new variants in the next commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New hooks return only their own message:
DataSource::try_to_proto_source, FileSource::try_to_proto_file_source
and DataSink::try_to_proto_sink. Built-ins implement only these, and
datafusion-proto always writes the layered form. The 55 hooks and the
whole-plan decode helpers are deprecated and otherwise unchanged.

The deprecated flat variants still decode: each is mapped to the
layered decoders, with format fields converted by the From impls in
datafusion-proto-models/src/physical_plan_compat.rs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@adriangb
adriangb force-pushed the proto-datasource-node-layer branch from a50ac0d to da361cb Compare October 27, 2026 23:10
@adriangb
adriangb marked this pull request as ready for review October 27, 2026 23:35
adriangb and others added 2 commits October 27, 2026 18:56
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@adriangb
adriangb force-pushed the proto-datasource-node-layer branch from da361cb to 009fa28 Compare October 27, 2026 23:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auto detected api change Auto detected API change datasource Changes to the datasource crate documentation Improvements or additions to documentation proto Related to proto crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants