Repository navigation
Conversation
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
adriangb
force-pushed
the
proto-datasource-node-layer
branch
from
October 27, 2026 20:18
748f78a to
56a3c63
Compare
This was referenced Oct 27, 2026
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Deprecated encoders now emit layered nodes that their paired decoders reject, breaking existing source and sink round trips and requiring decoder compatibility fixes and tests.
1 open issue
What changed in this PR
This PR introduces layered protobuf nodes that mirror DataFusion’s source and sink traits, preparing for the per-type decoder registries in #26555 and #26807.
Changes:
- Adds source, file-source, and sink messages and migrates built-in encoding to the layered format.
- Retains legacy decoding and deprecated hooks while centralizing compatibility conversions.
- Adds compatibility tests and documents migration and reader/writer version constraints.
| File | Description |
|---|---|
docs/source/library-user-guide/upgrading/56.0.0.md |
Documents deprecated hooks and wire-format migration. |
datafusion/proto/tests/cases/plans/sources.rs |
Updates source assertions for layered nodes. |
datafusion/proto/tests/cases/plans/sinks.rs |
Updates sink assertions for layered nodes. |
datafusion/proto/tests/cases/plans/mod.rs |
Registers compatibility tests. |
datafusion/proto/tests/cases/plans/legacy.rs |
Tests layered encoding, legacy decoding, and fallback behavior. |
datafusion/proto/src/physical_plan/mod.rs |
Dispatches layered and legacy source/sink decoding. |
datafusion/proto-models/src/physical_plan_compat.rs |
Maps legacy scan fields into source messages. |
datafusion/proto-models/src/lib.rs |
Exposes compatibility conversions. |
datafusion/proto-models/src/generated/prost.rs |
Adds generated layered protobuf types. |
datafusion/proto-models/proto/datafusion.proto |
Defines layered messages and retains legacy variants. |
datafusion/datasource/src/source.rs |
Adds source-level encoding and wrapper construction. |
datafusion/datasource/src/sink.rs |
Adds sink-level encoding and wrapper construction. |
datafusion/datasource/src/memory.rs |
Migrates memory-source serialization. |
datafusion/datasource/src/file.rs |
Adds file-source encoding with deprecated-hook forwarding. |
datafusion/datasource/src/file_scan_config/mod.rs |
Combines shared scan configuration with source messages. |
datafusion/datasource-parquet/src/source.rs |
Migrates Parquet source serialization. |
datafusion/datasource-parquet/src/sink.rs |
Migrates Parquet sink encoding. |
datafusion/datasource-json/src/source.rs |
Migrates JSON source serialization. |
datafusion/datasource-json/src/file_format.rs |
Migrates JSON sink encoding. |
datafusion/datasource-csv/src/source.rs |
Migrates CSV source serialization. |
datafusion/datasource-csv/src/file_format.rs |
Migrates CSV sink encoding. |
datafusion/datasource-avro/src/source.rs |
Migrates Avro source serialization. |
datafusion/datasource-arrow/src/source.rs |
Migrates Arrow source serialization. |
Review effort: Balanced
Adds data_source = 42 and data_sink = 43 to PhysicalPlanNode, with
DataSourceNode, FileSourceNode and DataSinkNode oneofs that each have an
extension {name, payload} arm. Generated code only; datafusion-proto
handles the new variants in the next commit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New hooks return only their own message: DataSource::try_to_proto_source, FileSource::try_to_proto_file_source and DataSink::try_to_proto_sink. Built-ins implement only these, and datafusion-proto always writes the layered form. The 55 hooks and the whole-plan decode helpers are deprecated and otherwise unchanged. The deprecated flat variants still decode: each is mapped to the layered decoders, with format fields converted by the From impls in datafusion-proto-models/src/physical_plan_compat.rs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
adriangb
force-pushed
the
proto-datasource-node-layer
branch
from
October 27, 2026 23:10
a50ac0d to
da361cb
Compare
adriangb
marked this pull request as ready for review
October 27, 2026 23:35
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
adriangb
force-pushed
the
proto-datasource-node-layer
branch
from
October 27, 2026 23:56
da361cb to
009fa28
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Which issue does this PR close?
ExecutionPlans to decode via a per-type registry instead ofPhysicalExtensionCodec#24631, feat: allow extensionPhysicalExprs to decode via a per-type registry #24628.Rationale for this change
The
DataSource,FileSourceandDataSinkproto hooks (added in 55.0.0) return a fullPhysicalPlanNode, which is the Exec wrapper. Decode in #26555 / #26807 works at the source level. So:extension_data_source_node); a source can return any plan variantFileSourceneeds a markerDataSourceThis PR adds a proto layer that mirrors the traits. Then #26555 / #26807 only add a decoder for the
extensionarms.Only the scan / sink variants of
PhysicalPlanNodeare shown. The otherExecutionPlanvariants (projection = 4,filter = 12, ...) do not change.Before:
After:
Timing: land before 56.0.0 (#24461), together with #26555 / #26807. Then the hooks added in 55.0.0 change only once, through deprecation.
What changes are included in this PR?
decode_file_scan,build_data_sink), by reference, with no subtree copy. Format fields map throughFrom<&XScanExecNode> for XSourceNodeindatafusion-proto-models/src/physical_plan_compat.rs: one mapping per format, exhaustive destructure. The deprecatedXSource::try_from_protohelpers use the same impls.Ok(None))DataSource::try_to_proto_source → DataSourceNodeFileSource::try_to_proto_file_source → FileSourceNodeDataSink::try_to_proto_sink → DataSinkNodetry_to_protohooks added in 55.0.0, and the old public decode functions. They are otherwise unchanged: the hook defaults still returnNone, and built-ins implement only the new hooks.PhysicalExtensionCodecextensionarmsNo decoder for extension DataSource '{name}'. The registry is in #26555 / #26807.Suggested shape for #26555 after this PR:
What is the testing strategy for this PR?
datafusion/proto/tests/cases/plans/deprecated_data_source_proto_serialization.rs:built_ins_encode_layered_and_legacy_variants_decodemissing_fields_give_clear_errors,extension_nodes_without_decoder_errorExisting scan and sink tests in
sources.rs/sinks.rsnow read the layered form.Feature flags:
cargo check -p datafusion-protopasses with no default features,avro, default and all features.-p datafusion-datasourceand-p datafusionpass with no default features.cargo test -p datafusion-proto --no-default-featuresdoes not compile, onmaintoo: existing tests import Parquet types without a gate.Are there any user-facing changes?
try_to_protohooks (added in 55.0.0) onDataSource,FileSourceandDataSink. Existing implementations still work. Migration notes are inupgrading/56.0.0.md.Open points
#[deprecated]on a trait method gives no warning to implementers who override it.protoxbecauseprotocwas not available. Rundatafusion/proto-models/regen.shto confirm.datafusion-proto(non-exhaustive match on the new variants). This is OK for a squash merge.🤖 Generated with Claude Code