Describe the bug
The Substrait consumer derives all three supported intersection schemas from the primary input's nullability. A field that the intersection makes required can therefore remain nullable in the logical output schema.
Reproduced on main at 1b6dc92a221b41088afa2753fe5526be6d057b7b.
The Substrait Set Operation rules define different nullability rules for each operation. With the spec's three input patterns (R = required, N = nullable), all fields having type i64:
primary: R R R R N N N N
secondary 1: R R N N R R N N
secondary 2: R N R N R N R N
the consumer returns:
| Operation |
Expected |
Actual |
| INTERSECTION_PRIMARY |
R R R R R N N N |
R R R R N N N N |
| INTERSECTION_MULTISET |
R R R R R R R N |
R R R R N N N N |
| INTERSECTION_MULTISET_ALL |
R R R R R R R N |
R R R R N N N N |
To reproduce
The plans and the table-registration probe are pinned in the conformance corpus. From a DataFusion checkout:
git clone https://github.com/alexandrefimov/substrait-conformance-cases conformance-cases
git -C conformance-cases checkout f12ba6103217fbd50b95c407b5dc3e8fae1e05a0
mkdir -p conformance-inputs datafusion/substrait/examples
cp conformance-cases/derived-schema/setop_*.json conformance-inputs/
cp conformance-cases/probe/datafusion_corpus_probe.rs datafusion/substrait/examples/corpus_probe.rs
SUBSTRAIT_CORPUS_DIR="$PWD/conformance-inputs" cargo run --locked -p datafusion-substrait --example corpus_probe
The probe registers empty named tables with exactly those input schemas and prints the logical schema immediately after from_substrait_plan, before optimization or execution. The setop_intersection_* lines show the three mismatches. Union and primary-minus controls match their respective spec rules; MINUS_MULTISET is explicitly unsupported.
Expected behavior
For multiset intersections, a field is required if any input requires it. For primary intersection, it is nullable only when it is nullable in the primary input and in at least one secondary input.
from_set_rel delegates to LogicalPlanBuilder::intersect, which uses a left semi join. Keeping the left field nullability there explains the observed schema. This report concerns the schema of consumed Substrait plans; it does not assert incorrect rows. The multiplicity bugs #12955 and #12956 are separate.
Describe the bug
The Substrait consumer derives all three supported intersection schemas from the primary input's nullability. A field that the intersection makes required can therefore remain nullable in the logical output schema.
Reproduced on
mainat1b6dc92a221b41088afa2753fe5526be6d057b7b.The Substrait Set Operation rules define different nullability rules for each operation. With the spec's three input patterns (
R= required,N= nullable), all fields having typei64:the consumer returns:
To reproduce
The plans and the table-registration probe are pinned in the conformance corpus. From a DataFusion checkout:
The probe registers empty named tables with exactly those input schemas and prints the logical schema immediately after
from_substrait_plan, before optimization or execution. Thesetop_intersection_*lines show the three mismatches. Union and primary-minus controls match their respective spec rules;MINUS_MULTISETis explicitly unsupported.Expected behavior
For multiset intersections, a field is required if any input requires it. For primary intersection, it is nullable only when it is nullable in the primary input and in at least one secondary input.
from_set_reldelegates toLogicalPlanBuilder::intersect, which uses a left semi join. Keeping the left field nullability there explains the observed schema. This report concerns the schema of consumed Substrait plans; it does not assert incorrect rows. The multiplicity bugs #12955 and #12956 are separate.