Describe the bug
The Substrait consumer maps standard decimal arithmetic to DataFusion operators whose derived precision and scale can differ from the referenced Substrait function definition. It accepts the plan and exposes the different type without reporting the mismatch.
Reproduced on main at 1b6dc92a221b41088afa2753fe5526be6d057b7b.
Each plan references extension:io.substrait:functions_arithmetic_decimal and declares the return type given by its YAML formula. The add, multiply and divide definitions are identical in spec v0.87.0, v0.102.0 and v0.103.0.
| Expression |
Spec / declared output_type |
DataFusion logical schema |
| dec(10,2) + dec(5,1) |
dec(11,2) |
dec(11,2) |
| dec(38,10) + dec(38,10) |
dec(38,9) |
dec(38,10) |
| dec(10,2) * dec(5,1) |
dec(16,3) |
dec(16,3) |
| dec(38,10) * dec(38,10) |
dec(38,6) |
dec(38,20) |
| dec(10,2) / dec(5,1) |
dec(21,8) |
dec(15,6) |
Inputs and declared outputs are required. The table focuses on precision and scale, with the two small add/multiply cases serving as controls.
To reproduce
From a DataFusion checkout:
git clone https://github.com/alexandrefimov/substrait-conformance-cases conformance-cases
git -C conformance-cases checkout f12ba6103217fbd50b95c407b5dc3e8fae1e05a0
mkdir -p conformance-inputs datafusion/substrait/examples
cp conformance-cases/derived-schema/decimal_*.json conformance-inputs/
cp conformance-cases/probe/datafusion_corpus_probe.rs datafusion/substrait/examples/corpus_probe.rs
SUBSTRAIT_CORPUS_DIR="$PWD/conformance-inputs" cargo run --locked -p datafusion-substrait --example corpus_probe
The probe registers a table with the exact declared input types. Its DATAFUSION ACCEPTED line reports the logical schema before execution. For example, decimal_divide declares decimal(21,8) and reports Decimal128(15, 6).
Expected behavior
The consumed expression should follow the return-type rule of the referenced decimal function, or reject a function contract it cannot implement. For division here, the YAML gives scale = max(6, 2 + 5 + 1) = 8 and precision = 10 - 2 + 5 + 8 = 21.
from_scalar_function resolves the function name to a native operator and constructs a BinaryExpr without checking its return type against the declaration. This is a Substrait conversion issue, not a request to change the semantics of ordinary DataFusion SQL arithmetic.
The wide-add fixture also reports an execution overflow at scale 10. The core reproducer is the schema mismatch: these plans omit overflow options, so this report does not prescribe a rounding or overflow policy. In particular, the wide-multiply input value can overflow even with the correct declared type.
Describe the bug
The Substrait consumer maps standard decimal arithmetic to DataFusion operators whose derived precision and scale can differ from the referenced Substrait function definition. It accepts the plan and exposes the different type without reporting the mismatch.
Reproduced on
mainat1b6dc92a221b41088afa2753fe5526be6d057b7b.Each plan references
extension:io.substrait:functions_arithmetic_decimaland declares the return type given by its YAML formula. The add, multiply and divide definitions are identical in spec v0.87.0, v0.102.0 and v0.103.0.Inputs and declared outputs are required. The table focuses on precision and scale, with the two small add/multiply cases serving as controls.
To reproduce
From a DataFusion checkout:
The probe registers a table with the exact declared input types. Its
DATAFUSION ACCEPTEDline reports the logical schema before execution. For example,decimal_dividedeclaresdecimal(21,8)and reportsDecimal128(15, 6).Expected behavior
The consumed expression should follow the return-type rule of the referenced decimal function, or reject a function contract it cannot implement. For division here, the YAML gives
scale = max(6, 2 + 5 + 1) = 8andprecision = 10 - 2 + 5 + 8 = 21.from_scalar_functionresolves the function name to a native operator and constructs aBinaryExprwithout checking its return type against the declaration. This is a Substrait conversion issue, not a request to change the semantics of ordinary DataFusion SQL arithmetic.The wide-add fixture also reports an execution overflow at scale 10. The core reproducer is the schema mismatch: these plans omit overflow options, so this report does not prescribe a rounding or overflow policy. In particular, the wide-multiply input value can overflow even with the correct declared type.