Describe the bug
The Substrait producer emits aggregate calls with output_type: None. The logical plan already has a known result type, but the exported call does not carry it. This affects count, sum, avg and min in the reproducer below.
Reproduced on main at 8a9228164fddeaa03c9ee510b641990e403f2eb5.
To reproduce
From a DataFusion checkout, using its pinned Rust toolchain:
git clone https://github.com/alexandrefimov/substrait-conformance-cases conformance-cases
git -C conformance-cases checkout c4d3ffd74842aa38079b3d523913199dc615a352
mkdir -p datafusion/substrait/examples
cp conformance-cases/probe/datafusion_producer_probe.rs datafusion/substrait/examples/corpus_producer.rs
cargo run --locked -p datafusion-substrait --example corpus_producer -- --aggregate-output-types
Use an unused example filename if corpus_producer.rs already exists. The probe registers an empty named table with a required Int64 column and calls to_substrait_plan for each aggregate. It inspects the produced protobuf directly, without converting it back through a consumer.
| Query |
Logical output type |
Exported AggregateFunction.output_type |
SELECT count(i) FROM t |
Int64 |
None |
SELECT sum(i) FROM t |
Int64 |
None |
SELECT avg(i) FROM t |
Float64 |
None |
SELECT min(i) FROM t |
Int64 |
None |
The final diagnostic is {"cases":4,"missing_output_types":4}.
Expected behavior
AggregateFunction.output_type must be set to the return type derived from the referenced function declaration. The producer should write a conforming type, or report that it cannot represent that function contract.
from_aggregate_function currently sets output_type: None explicitly. A consumer that validates required fields can reject these plans; one that uses the declaration for schema inference has no type to use.
This report is about the missing declaration. It does not prescribe an integer AVG return type or address aggregate phases. #24967 and #25045 concern phase handling in the consumer; #15831 fixed the corresponding missing-output-type issue for BinaryExpr.
Describe the bug
The Substrait producer emits aggregate calls with
output_type: None. The logical plan already has a known result type, but the exported call does not carry it. This affectscount,sum,avgandminin the reproducer below.Reproduced on main at
8a9228164fddeaa03c9ee510b641990e403f2eb5.To reproduce
From a DataFusion checkout, using its pinned Rust toolchain:
Use an unused example filename if
corpus_producer.rsalready exists. The probe registers an empty named table with a required Int64 column and callsto_substrait_planfor each aggregate. It inspects the produced protobuf directly, without converting it back through a consumer.SELECT count(i) FROM tSELECT sum(i) FROM tSELECT avg(i) FROM tSELECT min(i) FROM tThe final diagnostic is
{"cases":4,"missing_output_types":4}.Expected behavior
AggregateFunction.output_typemust be set to the return type derived from the referenced function declaration. The producer should write a conforming type, or report that it cannot represent that function contract.from_aggregate_functioncurrently setsoutput_type: Noneexplicitly. A consumer that validates required fields can reject these plans; one that uses the declaration for schema inference has no type to use.This report is about the missing declaration. It does not prescribe an integer AVG return type or address aggregate phases. #24967 and #25045 concern phase handling in the consumer; #15831 fixed the corresponding missing-output-type issue for BinaryExpr.