Is your feature request related to a problem or challenge?
Applications embedding DataFusion with their own TableProvider and ExecutionPlan implementations still pull in object_store when DataFusion's default features are disabled. These applications may provide their own file access and decoding, or use a different object_store version in their storage layer. They need a way to exclude DataFusion's storage dependency while retaining query planning and execution.
Describe the solution you'd like
Add a default-enabled object_store feature to the existing packages. Users should be able to disable default features and select SQL and expression features without including any object_store package in the enabled dependency graph.
The compatibility contract should be:
- Default builds retain existing storage behavior and APIs, including direct users of child crates such as
datafusion-execution and datafusion-physical-plan.
- Existing users with
default-features = false who need storage APIs explicitly enable object_store. Features that require built-in storage, such as parquet and avro, enable it automatically.
- Custom providers, in-memory tables, relational query execution, memory management, and local spilling remain available without the storage feature.
- No additional DataFusion packages are required.
Acceptance should use an independent downstream consumer: cargo tree --target all --edges all must contain no object_store package when storage is disabled, and a query using a custom provider and execution plan must run successfully. Explicitly restoring storage and using default features should preserve built-in file operations. Feature unification can still re-enable storage when another dependency requests it.
Describe alternatives you've considered
Disabling the existing default features does not remove the dependency. Aligning object_store versions can avoid duplicate versions but does not let an application exclude an unused storage implementation.
Additional context
Implementation: #25144.
Is your feature request related to a problem or challenge?
Applications embedding DataFusion with their own
TableProviderandExecutionPlanimplementations still pull inobject_storewhen DataFusion's default features are disabled. These applications may provide their own file access and decoding, or use a differentobject_storeversion in their storage layer. They need a way to exclude DataFusion's storage dependency while retaining query planning and execution.Describe the solution you'd like
Add a default-enabled
object_storefeature to the existing packages. Users should be able to disable default features and select SQL and expression features without including anyobject_storepackage in the enabled dependency graph.The compatibility contract should be:
datafusion-executionanddatafusion-physical-plan.default-features = falsewho need storage APIs explicitly enableobject_store. Features that require built-in storage, such asparquetandavro, enable it automatically.Acceptance should use an independent downstream consumer:
cargo tree --target all --edges allmust contain noobject_storepackage when storage is disabled, and a query using a custom provider and execution plan must run successfully. Explicitly restoring storage and using default features should preserve built-in file operations. Feature unification can still re-enable storage when another dependency requests it.Describe alternatives you've considered
Disabling the existing default features does not remove the dependency. Aligning
object_storeversions can avoid duplicate versions but does not let an application exclude an unused storage implementation.Additional context
Implementation: #25144.