Skip to content

Substrait producer omits the required output_type on aggregate functions #25049

Description

@alexandrefimov

Describe the bug

The Substrait producer emits aggregate calls with output_type: None. The logical plan already has a known result type, but the exported call does not carry it. This affects count, sum, avg and min in the reproducer below.

Reproduced on main at 8a9228164fddeaa03c9ee510b641990e403f2eb5.

To reproduce

From a DataFusion checkout, using its pinned Rust toolchain:

git clone https://github.com/alexandrefimov/substrait-conformance-cases conformance-cases
git -C conformance-cases checkout c4d3ffd74842aa38079b3d523913199dc615a352
mkdir -p datafusion/substrait/examples
cp conformance-cases/probe/datafusion_producer_probe.rs datafusion/substrait/examples/corpus_producer.rs
cargo run --locked -p datafusion-substrait --example corpus_producer -- --aggregate-output-types

Use an unused example filename if corpus_producer.rs already exists. The probe registers an empty named table with a required Int64 column and calls to_substrait_plan for each aggregate. It inspects the produced protobuf directly, without converting it back through a consumer.

Query Logical output type Exported AggregateFunction.output_type
SELECT count(i) FROM t Int64 None
SELECT sum(i) FROM t Int64 None
SELECT avg(i) FROM t Float64 None
SELECT min(i) FROM t Int64 None

The final diagnostic is {"cases":4,"missing_output_types":4}.

Expected behavior

AggregateFunction.output_type must be set to the return type derived from the referenced function declaration. The producer should write a conforming type, or report that it cannot represent that function contract.

from_aggregate_function currently sets output_type: None explicitly. A consumer that validates required fields can reject these plans; one that uses the declaration for schema inference has no type to use.

This report is about the missing declaration. It does not prescribe an integer AVG return type or address aggregate phases. #24967 and #25045 concern phase handling in the consumer; #15831 fixed the corresponding missing-output-type issue for BinaryExpr.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions